acceptodds
Under review as a conference paper at ICLR 2027

ThoughtTrace: Understanding User Thoughts in Real-World LLM Interactions

11 views

Abstract

Conversational AI has now reached billions of users, yet existing datasets capture only what people say, not what they think. We introduce *ThoughtTrace*, the first large-scale dataset that pairs real-world multi-turn human–AI conversations with users' self-reported thoughts: their reasons for sending prompts and reactions to assistant responses. *ThoughtTrace* comprises 1,058 users, 2,206 conversations, and 10,556 thought annotations collected across 20 language models. Our analysis shows that *ThoughtTrace* captures long-horizon, topically diverse interactions, and that thoughts are semantically distinct from messages, difficult for frontier LLMs to infer from context, and tied to the conversation stages. We further demonstrate the utility of thoughts for downstream modeling. First, thoughts improve user-behavior prediction as conditions for inference. Second, thought-guided rewrites provide fine-grained alignment signals for training personalized assistants. Together, *ThoughtTrace* establishes user thoughts as a new data modality for studying the thought dynamics behind human–AI interaction and provides a foundation for building assistants that better understand and adapt to users' latent goals, preferences, and needs.

Primary area:
datasets and benchmarks
Keywords:
User Thoughts, Human-AI Interaction, User Modeling, Alignment, Personalization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_FVXEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ThoughtTrace, a dataset pairing multi-turn human–AI conversations with users' self-reported thoughts (reasons for messages and reactions to responses), and shows their use for next-message prediction and alignment.

Critical strengths
  • User thoughts collected during conversations are a new and useful type of data, especially for user modeling and simulation.
  • The collection protocol is carefully designed, and the paper is clearly written.
Critical weaknesses
  • The prediction experiment provides limited evidence for the utility of thoughts. The input includes the user's reaction to the previous response, which often already states the intent of the next message, so the gain is largely expected and mainly reflects correlation; such thoughts are also unavailable at deployment.
  • The alignment experiment does not clearly isolate the effect of thoughts. All chosen responses are GPT-5.4 rewrites, so much of the gain may come from distillation, and the thought- vs. message-guided comparison also changes how dissatisfied responses are selected.
  • Arena-Hard measures general response quality without user context, so it does not directly support the claim about personalized assistants.
  • With the help of an LLM, I scanned the released dataset: about one third of the thoughts were written after the conversation ended, most reactions are generic (e.g., "satisfied", "good"), and many reasons are templated (e.g., "conversation starter"). Without a human baseline, the claim that thoughts are hard for LLMs to infer may reflect this annotation noise.
Questions for authors

see above

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer FVXE for recognizing user thoughts as a new and useful type of data for user modeling and simulation, the carefully designed collection protocol, and the clear writing, and for inspecting the released dataset. Below, we add controls and ablations to the prediction experiment and analyze annotation timing and quality.

W1. Reactions may preview the next message, and thoughts may be unavailable at deployment.

Our analysis in Sec. 4.2 shows that reactions typically do not state the intent of the next message directly, and our additional experiments show that most of the prediction gain remains without the immediate reaction. Thoughts also need not be available at deployment, since they can serve as training data.

(1) Reactions typically do not state the intent of the next message directly. In a shared UMAP projection of text embeddings, reaction–next-message pairs are much farther apart than consecutive user messages (median distance 6.62 vs. 1.96; centroid distance 0.320 vs. 0.120; MMD 0.257 vs. 0.096; App. B.9, see lines 1368–1400). An LLM judge rates the next message's coverage of the reaction as minimal (2.00/5; see line 305), and frontier LLMs that are given the next message still recover the reaction poorly (2.54/5; see line 315). App. B.3 (see line 1140) also shows a case where an accurate reaction does not determine what the user says next.

(2) The gain does not rely on the immediate reaction. We run an additional experiment with three ablations of the thought-augmented context:

Table. Ablations of the thought-augmented context in next-message prediction: semantic similarity to the user's actual next message (0–100) and relative gain over history-only.

ContextSimilarity (0–100)Relative gain
History-only19.3–
+ Reasons only24.4+26%
+ Reactions only25.9+34%
+ All thoughts except the reaction to the most recent response24.8+28%
+ All thoughts (thought-augmented)28.1+46%

Most of the gain therefore remains without the reaction that could preview the next message, including from reasons and earlier-turn thoughts. We have included these ablations in the updated Sec. 5.1 (lines 442–447) and App. B.14 (Table 9).

(3) Uses of collected thoughts.

  • Uses that do not need thoughts at deployment. As an extension of Sec. 5.1, where thoughts improve prediction of the user's next message (+8.4 points across six models, Table 1, see line 437), the collected thoughts can be summarized into user personas to build better user simulators. Thoughts also serve as additional training data and are not needed at inference: the model trained with thought-guided DPO runs without thoughts and outperforms message-guided DPO on Arena-Hard (+4.5 style-controlled win, Table 2, see lines 478–481).

  • Future direction. How to collect a minimal set of critical thoughts from users at deployment, to complement other signals for continual model improvement, is a promising direction beyond our current scope. Our contribution is to show that thoughts improve user behavior simulation and model alignment, and to release ThoughtTrace as a data source for future research.

W2. The alignment gain may partly come from GPT-5.4 distillation and from how dissatisfied responses are selected.

All trained conditions share the same distillation, and the advantage of thoughts holds when selection is controlled.

(1) Distillation is shared by all trained conditions. The WildChat, ThoughtTrace (messages), and ThoughtTrace (thoughts) conditions all use GPT-5.4 rewrites as chosen responses, with the same rewriter, base model, and DPO hyperparameters (App. D.9, see lines 2630–2646 and lines 2710–2713). The gain over the base model (22.5 → 41.5–48.1 style-controlled; see lines 478–481) therefore includes learning from GPT-5.4 rewrites, and the thought-specific effect is the gap between rewrite conditions: +3.9 win and +4.5 style-controlled win over message-guided rewrites on the same conversations. We have included this distinction in the updated Sec. 5.2 (lines 469–471).

(2) Selection vs. guidance. The main comparison varies both which responses are selected and how they are rewritten, and selection is part of the value of thoughts: on the same conversations, thoughts surface 1,000 dissatisfaction instances while messages surface 450 (see line 483). Two existing results address this. At matched size (450 instances), thought guidance still leads by +3.1 win / +3.7 style-controlled (App. B.5, see lines 1184–1186). In the human evaluation (App. B.4, see lines 1147–1163), both rewrites are generated for the same dissatisfaction instances, so selection is held fixed, and the thought-guided rewrite is preferred in 70.0% of the 100 pairwise comparisons (20 cases × 5 annotators) and wins the majority vote in 16 of 20 cases.

AuthorsOpenReview ↗

Rebuttal by Authors

W3. Personalized assistance can be evaluated more directly than with Arena-Hard.

Arena-Hard tests whether thought supervision improves general response quality, i.e., whether the gains hold beyond ThoughtTrace users. For user-specific evaluation, App. B.4 (see lines 1147–1163) uses 20 held-out ThoughtTrace dissatisfaction cases with five annotators each, who judge rewrites against the original user's own thought. Thought-guided rewrites better address that user's dissatisfaction in 70.0% of the 100 pairwise comparisons (20 cases × 5 annotators) and win the majority vote in 16 of 20 cases. We have included this clarification in the updated Sec. 5.2 (lines 468–469).

W4. Annotation timing and brief or templated thoughts may affect the inference-difficulty result.

Thank you for inspecting the released data closely. This level of noise is common in real-world data, most thoughts are specific to their turn, and our results do not rely on late or low-information thoughts.

(1) This level of noise is common in real-world data collection. LMSYS-Chat-1M reports "a significant amount of low-quality and duplicate data" and applies no filtering on purpose "to reflect the real-world distribution" (Zheng et al., 2023). In WildChat, many jailbreaking prompts are exact copies circulating online; the most common of them appears 3,903 times, from 211 users (Zhao et al., 2024b). Similar noise does not prevent ThoughtTrace from being informative, as our experiments and the analyses below show.

(2) Late thoughts are by design and do not drive the gain.

  • Late annotation is by design and mostly covers the final turn. The reviewer's count matches ours: 34.8% of thoughts were written after the conversation's last message. Post-conversation annotation was allowed by design (App. C.3–C.4, see lines 1783–1784, 1821), so that participants could finish recording thoughts after the 10-minute window. We run an additional analysis of the timestamps: most of these thoughts annotate the final turn, which no prediction target follows, the median delay between a message and its thought is about one minute, and 90% are written within eight minutes.

  • Late thoughts do not drive the prediction gain. We also run an additional experiment that repeats next-message prediction while controlling for when thoughts were written: thoughts improve prediction over history-only by +47% on instances where every thought was written before the target message, matching the gain on all instances (+46%). We have included these analyses and results in the updated App. B.12 and B.14.

(3) Very short and templated thoughts are uncommon. We run an additional analysis of thought content. Only 22.6% of reactions have three words or fewer, mostly explicit approvals such as "good". These are genuine satisfaction signals, which we keep rather than distort the natural distribution. Templated reasons are rare: only 4.3% of reasons repeat another reason's exact text, and "conversation starter" appears in 5 of 4,498. We have included these statistics in the updated App. B.12.

(4) Most thoughts are specific to their turn, not generic or templated. We run an additional experiment that randomly samples 100 reasons and 100 reactions and rates each on a 1–5 informativeness scale (1 = empty, 2 = generic, 3 = specific to the turn, 4 = adds information beyond the message, 5 = rich personal context). 74% of reasons and 56% of reactions are specific to the turn (≥3), and 27% of each add information beyond the message (≥4). We have included these ratings in the updated App. B.12. Neither utility experiment in Sec. 5 relies on low-information thoughts: Sec. 5.1 keeps examples whose thoughts are rated ≥4/5 (see line 2522), and Sec. 5.2 uses only dissatisfaction reactions of at least six words (see line 2625).

(5) Low-information or retrospective thoughts do not drive the inference result. We run an additional experiment that repeats Thought Property 2 by thought informativeness and timing. Informative thoughts (rated ≥4/5) are harder to infer than the average sampled thought (reasons 2.85 vs. 3.17; reactions 2.70 vs. 2.96), and thoughts written before the next user message are about as hard (3.22 and 2.85). We have included these results in the updated Sec. 4.2 (lines 316–319) and App. B.13.

(6) Human validation confirms that the thoughts are informative. Because a message admits many plausible thoughts, low inference scores reflect the gap between an observer's guess and the user's ground truth rather than noise in the reports. As a human check, 30 independent raters find that the thoughts add information beyond the messages (4.0/5) and explain user behavior (4.2/5, App. B.1, see lines 306–308, 763–770). The thoughts also improve prediction of the actual next message (+8.4 points across six models; see line 437), and humans prefer the rewrites they guide (70.0%; see lines 1162–1163).

AuthorsOpenReview ↗

Official Comment by Authors

We hope that our responses and additional experiments conducted during the rebuttal period have adequately addressed your questions! We would be happy to address any further questions or suggestions you may have, and we would greatly appreciate it if you could consider increasing your score if you find our responses helpful!

Reviewer_k8fvOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

ThoughtTrace is a dataset pairing real multi-turn human–LLM conversations with users’ self-reported thoughts (reasons for their messages, reactions to responses). It has about 1,058 Prolific participants, 2,155 conversations and 10,174 thought annotations across 20 models. The authors show that thoughts differ from messages, are hard for frontier LLMs to infer, fall into distinct types, and shift across conversation stages. Two experiments are offered as validation: thoughts improve next-message prediction, and finetuning on thought-guided rewrites beats message-guided rewrites on Arena-Hard.

Critical strengths

The idea is new and well motivated. Existing datasets only capture what users say, so first-person thoughts open a direction that has barely been explored and could be reused for user modelling and alignment. The data resembles real everyday AI use. Conversations are longer than in WildChat and LMSYS-Chat, topics are broad, and demographics match frequent AI users. The properties in Section 4 are laid out clearly.

Critical weaknesses

Thoughts are treated as clean, generalisable supervision, but they depend on the user’s demographics, context and mental state, and users may weigh several possibilities before settling on one. Each context has a single annotator, and annotation noise is not examined. The true overlap with messages is probably lower than reported, and noisy thoughts could disrupt alignment training. Having multiple users annotate the same context would help. The downstream validation is narrow. Table 1 tests only next-message prediction, which does not necessarily translate into better signals for building a collaborative agent or may lead to overfitting user intent. The LLM-based labelling of topics and thought types is also not validated, so it is unclear how these properties were annotated.

Questions for authors

What was the cost incurred to recruit 1058 users to build ThoughtTrace?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer k8fv for recognizing first-person thoughts as a new and well-motivated direction for user modeling and alignment, the resemblance of our data to everyday AI use, and the clear analysis in Sec. 4. Below, we address annotation reliability and the scope of our downstream evaluation, including new robustness checks and a human validation of the LLM-based labels.

W1. Annotation noise may affect thoughts as supervision, and multiple annotators per context may help.

Thoughts are first-person reports, so other annotators can only infer them. Their noise is limited and does not disrupt alignment training.

(1) Thoughts are first-person reports, so a second annotator cannot label the same thought. A user's reason or reaction depends on private goals, constraints, and interpretations that others cannot observe. Other annotators of the same context would have to infer it, and each could produce a different, equally plausible thought; their disagreement would measure how hard the thought is to infer, not noise in the user's own report. The self-report is the ground truth, which is why we adopted it, and Thought Property 2 shows that even frontier LLMs cannot recover it reliably (see line 315). Thoughts do depend on the user's demographics, context, and mental state. This variation is what ThoughtTrace is designed to capture, and we release demographics, task descriptions, and timestamps (see lines 1951–1961) so that it can be modeled. We do not claim thoughts are user-independent labels, and we have included this clarification in the updated Sec. 3.1 (lines 151–153). Users who weigh several possibilities can attach multiple reasons or reactions to one message (App. C.4, see line 1836).

(2) Noise is common in real-world data collection. Prior datasets of real user conversations contain substantial noise. LMSYS-Chat-1M reports "a significant amount of low-quality and duplicate data" and applies no filtering on purpose "to reflect the real-world distribution" (Zheng et al., 2023). In WildChat, many jailbreaking prompts are exact copies circulating online; the most common of them appears 3,903 times, from 211 users (Zhao et al., 2024b). Similar noise in ThoughtTrace does not prevent it from providing sufficient information, as our experiments and the analyses below show.

(3) Existing safeguards against noise. Collection includes a tutorial with a comprehension quiz, pilot-informed interface design, speed-based exclusion, and post-collection cleaning (App. C.7–C.8, see lines 1903–1929). In an independent human study, 30 Prolific raters agreed that thoughts contain information beyond the messages (4.0/5) and help explain user behavior (4.2/5), with a per-conversation SD of 0.2 (App. B.1, see lines 763–770). Both experiments use filtered thoughts: Sec. 5.1 keeps examples whose thoughts an LLM judge rates ≥4/5 for informativeness (see line 2522), and Sec. 5.2 uses dissatisfaction reactions of at least six words (see line 2625).

(4) Our results are robust to annotation noise.

  • Noise would not produce our downstream gains. Pure noise would not improve prediction of the user's actual next message (+8.4 points on average across six models, Table 1, see line 437), nor would it guide rewrites that human annotators prefer: on 20 held-out dissatisfaction cases with five annotators each, thought-guided rewrites are preferred over message-guided rewrites in 70.0% of 100 comparisons and win the majority vote in 16 of 20 cases (Sec. 5.2; Table 5, App. B.4; see lines 1154–1163).

  • Results hold for informative and pre-message thoughts. We run an additional experiment that repeats Thought Properties 1–2 by thought informativeness and timing. Informative thoughts (rated ≥4/5) are harder to infer than the average sampled thought (1–5 similarity: 2.85 vs. 3.17 for reasons; 2.70 vs. 2.96 for reactions). Messages also cover informative reasons less (1–5 coverage: 2.70 vs. 3.47), and reactions stay minimally covered (2.25 vs. 1.79). Thoughts written before the next user message give the same picture (coverage 3.47 and 1.84; inference 3.22 and 2.85). We have included these results in the updated Sec. 4.2 (lines 316–319) and App. B.13.

(5) Noise did not disrupt alignment training. On the same conversations, thought-guided DPO outperforms message-guided DPO (47.9 vs. 44.0 win; 48.1 vs. 43.6 style-controlled; see lines 478–481), including at matched training size (47.1 vs. 44.0, App. B.5, see lines 1184–1186).

AuthorsOpenReview ↗

Rebuttal by Authors

W2. Downstream validation could be broadened.

Thoughts also improve a general assistant beyond next-message prediction.

(1) Our experiment in Sec. 5.2 trains a general assistant, and its gains are not tied to ThoughtTrace users. Beyond Table 1, Sec. 5.2 uses thoughts only as training supervision (see lines 460–468). The DPO-trained model never sees thoughts and is evaluated on Arena-Hard, whose prompts are unrelated to ThoughtTrace users. Thought supervision improves over message supervision on the same conversations (+3.9 win / +4.5 style-controlled; see lines 478–481), also at matched size (+3.1 / +3.7, App. B.5, see lines 1184–1186). This suggests it improves general assistant quality rather than overfitting to specific users' intents. Human annotators also prefer thought-guided rewrites on held-out dissatisfaction cases (70.0%, App. B.4, see lines 1162–1163).

(2) Scope of the prediction result. Next-message prediction is the standard training and evaluation target for user simulators (Naous et al., 2025; Abdulhai et al., 2025; Wu et al., 2026; see lines 449–450), which are in turn used to train and evaluate collaborative assistants. Prediction alone does not establish a better collaborative agent, which is why Sec. 5.2 also tests thoughts as training supervision for an assistant.

W3. The LLM-based labels are not validated, and it is unclear how they were annotated.

Following your suggestion, we conduct additional human validation to support the reliability of the LLM-based labels.

(1) How the labels are produced and used. Topics are labeled by GPT-5.4 (temperature 0) as multi-label assignments from a predefined taxonomy of 36 subtopics, discarding any out-of-taxonomy labels (App. D.2, see lines 2068–2073). Reason types are labeled with the conversation context, and reaction types with the assistant response being reacted to (App. D.6, see lines 2365–2372); all prompts are in the appendix. These labels support the descriptive analyses (Fig. 3b, Thought Properties 3–4), and the dissatisfaction labels select training instances in Sec. 5.2; Thought Properties 1–2 and Table 1 do not use them.

(2) Human validation. An expert human annotator independently re-labels a random sample of 200 reasons and 200 reactions, using the same prompts and remaining blind to the original GPT-5.4 labels. Agreement with the GPT-5.4 labels is 89.0% for reaction types (Cohen's κ = 0.72) and 93.5% for the satisfied-vs-dissatisfied split used to select training instances (κ = 0.77). Reason types achieve 67.0% agreement (κ = 0.59), with most disagreements occurring between task-level categories (Task Motivation, Task Continuation) and Context Grounding or Content Expectation, which often co-occur within a single thought. We have included this validation in the updated Sec. 4.2 (lines 367–371) and App. D.6 (lines 2458–2473).

Q1. The cost of recruiting 1,058 users to build ThoughtTrace.

Building ThoughtTrace took approximately six months and cost about $6,000 in total (App. C.9).

AuthorsOpenReview ↗

Official Comment by Authors

We hope that our responses and additional experiments conducted during the rebuttal period have adequately addressed your questions! We would be happy to address any further questions or suggestions you may have, and we would greatly appreciate it if you could consider increasing your score if you find our responses helpful!

Reviewer_K46oOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ThoughtTrace, a dataset that pairs multi-turn human–AI conversations with users’ self-reported thoughts. To collect this data, the authors recruit diverse participants on Prolific to complete two self-defined tasks while optionally annotating their reasons for sending messages and reactions to assistant responses, yielding 2,155 conversations with over 17k messages and 10k thought annotations across 20 language models. Through analyses, the authors reveal that the annotated thoughts (1) convey information not verbalized in the messages and (2) remain challenging for current frontier LLMs to infer from the conversation alone. To study the downstream utility of the dataset, the authors first show that providing annotated thoughts at inference time improves prediction of the next user message across six LLMs. They further leverage dissatisfaction reactions to construct preference pairs for improving model response generation via DPO, reporting higher Arena-Hard win rates for Qwen3.5-4B.

Critical strengths
  • The dataset is collected through diverse interactions between human participants and 20 different LLMs on tasks that the participants define themselves, with free-text reasons and reactions written by the users during the conversation. This provides a resource for studying how users’ stated intentions and response evaluations relate to their conversational behavior.

  • The paper is well documented. The appendix provides details of the data collection protocol, annotation interface, and prompts used in experiments, helping readers assess and reuse the dataset.

  • The dataset is collected with explicit participant consent, and the authors also report excluding direct personal identifiers from the dataset to protect participant privacy.

Critical weaknesses
  • A major concern of this work lies in the lack of adequate positioning with respect to prior work, which makes it difficult to assess the novelty of the contribution. ThoughtTrace is closely related to an established line of research on free-text inference annotations in dialogue. CICERO [1] already includes utterance-level motivation and emotional reaction annotations (conceptually similar to the reasons and reactions in ThoughtTrace), while Reflect [2] annotates implicit information in dialogue and studies its use in response generation. These connections are not discussed, even though they seem central to the problem the paper addresses. As a result, it remains unclear whether the contribution lies primarily in the source and setting of data collection or represents a broader advance in capturing users’ latent thoughts. Without a more thorough discussion, its characterization as a “new paradigm” remains insufficiently justified.

References

[1] CICERO: A Dataset for Contextualized Commonsense Inference in Dialogues (Ghosal et al., ACL 2022)

[2] Reflect, Not Reflex: Inference-Based Common Ground Improves Dialogue Response Quality (Zhou et al., EMNLP 2022)

  • The next-user-message prediction experiment provides insufficient evidence for the distinctive utility of self-reported thoughts. The evaluation compares thought-augmented prediction only against a dialogue-history-only baseline, without considering alternative utterance-level annotations, such as motivations and reactions inferred by human annotators or LLMs from the same dialogue history. Without these controls, the reported gains cannot be attributed specifically to the information captured by participant self-reports. The experiment therefore leaves unresolved whether these annotations provide deeper insight into users’ latent mental states than existing forms of utterance-level contextual annotation.

  • The evaluation of next-user-message prediction raises concerns about potential temporal information leakage. The collection protocol allows participants to add thoughts at any time, including after the task timeout (Appendices C.3 and C.4), but the prediction setup does not specify whether annotations are restricted to those recorded before the target user message. An annotation associated with an earlier turn may have been added retrospectively and could therefore incorporate knowledge of subsequent messages. This leaves open the possibility that the reported gains partly reflect access to hindsight information rather than the ability to anticipate user behavior.

  • A further concern is whether the preference-training experiment adequately supports the claim of improved alignment with users’ preferences. Higher Arena-Hard win rates indicate better performance under a general response-quality evaluation, but do not directly establish that the trained model better satisfies the specific intentions and preferences expressed in users’ thoughts. While a human evaluation is conducted to complement the Arena-Hard evaluation, it compares the GPT-generated rewrites rather than the outputs of the DPO-trained model and therefore does not directly validate the claimed improvement in alignment with users’ preferences.

Questions for authors

See Weaknesses

7 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank Reviewer K46o for recognizing ThoughtTrace as a well-documented resource of diverse interactions with 20 LLMs, annotated with users' own reasons and reactions and collected with explicit consent and privacy protection. Below, we clarify how self-reports differ from inferred annotations, add inferred-thought controls to the prediction experiment, and test for temporal leakage.

W1. Relation to free-text inference annotations in dialogue (CICERO, Reflect).

Thank you for these pointers. Unlike CICERO [1] and Reflect [2], where third-party annotators infer implicit information in dialogues between people, ThoughtTrace records the thoughts that users themselves report while interacting with LLMs. We now discuss both works in the updated introduction (Sec. 1, lines 45–48) and related work (Sec. 2, lines 117–122).

(1) While CICERO and Reflect annotate what an observer can plausibly infer, ThoughtTrace records what the user reports thinking.

  • CICERO and Reflect collect an observer's inference. In CICERO, annotators read a dialogue and write what it suggests or, when it gives no answer, guess a plausible scenario [1]. In Reflect, crowd workers answer questions such as "what is the speaker feeling right now?" from the respondent's point of view, and Reflect itself describes CICERO-style annotation as "human inferences after reading the whole dialogue as a bystander" [2].

  • ThoughtTrace collects the user's own report. In ThoughtTrace, the user who sent the message reports their own reason or reaction during the conversation, attached to the turn and never shown to the AI (see line 1836). Since a message admits many plausible thoughts, an observer's inference is one guess among several; only the user's report gives the ground truth, including personal goals, constraints, and judgments the dialogue never shows.

(2) The gap between the two is large, and we measure it.

  • Thoughts are hard to infer from the dialogue. Thought Property 2 tests exactly this inference: given the same context, three frontier LLMs reach only 2.93/5 similarity to self-reported reasons and 2.54/5 to reactions, even when shown the next message (see line 315). Users' own messages cover their reasons only partially (3.22/5) and their reactions minimally (2.00/5; see lines 304–305). Inference datasets thus teach plausible inferences, while ThoughtTrace provides the first-person ground truth to evaluate and train them against.

  • Inferred thoughts barely help prediction. In an additional experiment, replacing self-reports with LLM-inferred thoughts at the same positions barely improves next-message prediction (history-only 19.3 → 20.8), while self-reports reach 28.1 (details in W2). Self-reports thus cannot be replaced by LLM-inferred pseudo-annotations.

(3) ThoughtTrace captures users' thoughts in human–LLM interaction rather than in existing dialogues between people from specific settings.

  • Dialogue sources. CICERO and Reflect both annotate such dialogues; CICERO's, for example, come from English-learning materials and exams (DailyDialog, MuTual, DREAM). Reflect uses its inferences to improve model responses, but none of its annotations involve a person interacting with an AI. ThoughtTrace instead records 2,155 conversations with 20 LLMs on self-defined tasks (see line 52).

  • Insight into human–LLM interaction. As a result, ThoughtTrace reveals why users phrase a request to an LLM as they do and how they judge its response. For example, reactions identify which responses are unsatisfactory and why, which enables our alignment experiment (Sec. 5.2): thought-guided rewrites outperform rewrites guided by the user's follow-up message (48.1 vs. 43.6 style-controlled win rate; see lines 478–481) and are preferred by human annotators in 70.0% of comparisons (App. B.4, see lines 1162–1163).

(4) Summary of differences.

Table. How CICERO, Reflect, and ThoughtTrace collect annotations.

CICEROReflectThoughtTrace
AnnotatorThird-party annotatorsCrowd workers (AMT)The user who wrote the message
PerspectiveObserver's inference, possibly a guessInference from the other speaker's point of viewThe user's own report
TimingAfter reading the full dialogueAfter reading the dialogue contextDuring the user's own conversation
DialoguesHuman-written dialogues between people (DailyDialog, MuTual, DREAM)600 contexts from human-written dialogues between people2,155 human–LLM conversations with 20 LLMs on self-defined tasks
Annotated contentSelected utterances: cause, subsequent event, prerequisite, motivation, listener's emotionSelected turns: speaker traits and emotion, responder emotion, what happened before or afterBoth sides of the interaction, at any turn: why the user sent each message (reasons) and how they judged each LLM response (reactions)
AuthorsOpenReview ↗

Rebuttal by Authors

W2. The prediction experiment can be strengthened with controls using inferred annotations.

Such controls separate the value of self-reported thoughts from the value of adding any turn-level annotation to the context. Our experiments show that inferred thoughts are inaccurate and fail to replicate the gains achieved with self-reported thoughts.

(1) Inferred thoughts are inaccurate, and are fundamentally different from self-reported thoughts. Thought Property 2 shows that LLM-inferred thoughts diverge substantially from self-reports: given the same context, three frontier LLMs reach only 2.93/5 similarity for reasons and 2.54/5 for reactions, even when shown the user's next message (see line 315). This is expected. An annotation inferred from the dialogue history, whether by a human or an LLM, contains no information about this particular user beyond what the history already gives the predictor, and different observers may infer different, equally plausible thoughts for the same message. The user's self-report is the ground truth and carries personal context that the history lacks.

(2) Inferred annotations do not reproduce the gain. We run an additional experiment that replaces every self-reported thought with an LLM-inferred one at the same position, generated from the dialogue history only (App. D.5 prompts, see lines 2289–2311), and keeps everything else fixed. Inferred thoughts give only a small gain that is not statistically significant (history-only 19.3 → 20.8, +8%), whereas self-reported thoughts reach 28.1 (+46%); the gap between self-reported and inferred thoughts is significant (95% CI of the difference: [1.9, 12.9]). We have included these in the updated Sec. 5.1 (lines 442–447) and App. B.14 (Table 9).

The alignment experiment already includes a similar control, in which rewrites are guided by the user's observable follow-up message instead of thoughts. Thought-guided rewrites outperform them on Arena-Hard (+3.9 win, +4.5 style-controlled win; see lines 478–481) and in a human evaluation (70.0% preference, App. B.4, see lines 1162–1163).

W3. Thoughts added after the target message may carry hindsight information.

Thank you for this careful observation. Participants could add thoughts after the task timeout, so that they could finish recording their thoughts rather than lose them at the 10-minute cutoff. Since every message and thought in ThoughtTrace is timestamped, we run additional experiments and find that the gain does not rely on hindsight.

(1) Most thoughts used for prediction were written before the message being predicted. We run an additional analysis of the per-thought timestamps. The median delay between a message and its thought is about one minute, and 90% of thoughts are written within eight minutes. Although 34.8% of thoughts were written after the conversation's last message, most of these annotate the final turn, which no prediction target follows. Overall, 77.3% of the thoughts that enter prediction contexts were written before the target message. We have included these statistics in the updated App. B.12 and B.14, with pointers in Sec. 3.2 (line 186) and App. C.4.

(2) The gain holds without post-target thoughts. We run an additional experiment that repeats next-message prediction while controlling for when thoughts were written. On instances where every thought was written before the target message (72% of instances), thoughts improve prediction from 18.8 to 27.6 (+47% relative), matching the gain on all instances (+46%). We have included these results in the updated Sec. 5.1 (lines 442–447) and App. B.14.

(3) Reactions are not hindsight paraphrases of the next message. Our existing analysis (Thought Property 1, see lines 296–304) compares each reaction with the user's next message: if a reaction merely restated that message, the two would be close in meaning. They are not. In a shared UMAP projection of text embeddings, a reaction lies much farther from its next message than consecutive user messages lie from each other (median distance 6.62 vs. 1.96; see lines 1368–1379), and a linear classifier on the embeddings separates reactions from next messages almost perfectly (AUC 0.988 vs. 0.721 for consecutive messages; App. B.9, see lines 1396–1400). An LLM judge that scores how well the next message expresses the reaction (1 = no overlap, 5 = full coverage) gives only 2.00/5 on average, i.e., minimal overlap (App. D.4; see lines 304–305).

AuthorsOpenReview ↗

Official Comment by Authors

We hope that our responses and additional experiments conducted during the rebuttal period have adequately addressed your questions! We would be happy to address any further questions or suggestions you may have, and we would greatly appreciate it if you could consider increasing your score if you find our responses helpful!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response by Authors

We thank all reviewers for their constructive feedback. Reviewers found that first-person thoughts are a new and well-motivated data type for user modeling, simulation, and alignment (k8fv, FVXE), that ThoughtTrace captures diverse, everyday interactions with 20 LLMs (K46o, k8fv) under a careful protocol with explicit consent and privacy protection (K46o, FVXE), and that the paper is clear and well documented (K46o, k8fv, FVXE). We have carefully addressed each of your concerns, and this response outlines the updates made to our manuscript.

(1) Self-reported thoughts differ from inferred annotations. Unlike CICERO and Reflect, where third-party annotators infer implicit information in dialogues between people, ThoughtTrace records the thoughts that users themselves report while interacting with LLMs (see updated Sec. 1, lines 45–48, and Sec. 2, lines 117–122). Replacing self-reports with LLM-inferred thoughts at the same positions barely improves next-message prediction (19.3 → 20.8), whereas self-reports reach 28.1. We have included this experiment in the updated Sec. 5.1 (lines 442–447) and App. B.14.

(2) The prediction gain does not rely on hindsight or on reactions that preview the next message. 77.3% of the thoughts in prediction contexts were written before the target message. On instances where every thought precedes the target, thoughts still improve prediction by +47% (vs. +46% on all instances), and removing the reaction to the most recent response still leaves a +28% gain. We have included these results in the updated Sec. 5.1 (lines 442–447) and App. B.12 and B.14.

(3) Thoughts and labels are reliable. As first-person reports, thoughts are user-specific by design rather than user-independent labels (see updated Sec. 3.1, lines 151–153). Most thoughts are specific to their turn (74% of reasons, 56% of reactions), and informative thoughts are even harder for LLMs to infer (2.85 vs. 3.17 for reasons; 2.70 vs. 2.96 for reactions). An expert annotator's blind re-labeling agrees with the LLM labels on 89.0% of reaction types (κ = 0.72), 93.5% of the satisfied-vs-dissatisfied split used for training (κ = 0.77), and 67.0% of reason types (κ = 0.59). We have included these results in the updated Sec. 4.2 (lines 316–319 and 367–371) and App. B.12, B.13, and D.6.

(4) The alignment gain comes from thoughts, not only from distillation. All trained conditions share the GPT-5.4 rewriter and DPO setup, so the effect of thoughts is the gap between rewrite conditions (+4.5 style-controlled win), which holds at matched training size and in a human evaluation of individual users' dissatisfaction (70.0% preference) (see updated Sec. 5.2, lines 468–471).

Finally, we share additional evidence of the utility of ThoughtTrace. Since its public release, ThoughtTrace has been widely adopted by independent works for diverse purposes beyond our experiments: to train user simulators with thought supervision, to improve and evaluate user-simulation fidelity, to evaluate whether models can recover users' latent reasons for personalization, to analyze reliance on LLMs across demographic groups, and to construct benchmarks. One independent user-simulator work reports that adding ThoughtTrace thought supervision to its training pipeline raises the simulator's accuracy from 36.0 to 63.0 on a role-play task that tests whether it can faithfully reproduce common student errors in K-12 math. We do not cite these works here to preserve anonymity under the double-blind policy.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.