acceptodds
Under review as a conference paper at ICLR 2027

GoalBench: Benchmarking goal inference in multi-turn interactions

Abstract

Our objective is to measure the ability of AI systems to infer the goals of human users in naturalistic, multi-turn interactions, a crucial step toward the development of controllable AI systems. However, standardized evaluation remains a challenge due to the difficulty of collecting ground truth data, and scalably assessing model outputs against this ground truth. To address this, we developed a methodology for annotation and evaluation, which uses third-person goal inferences provided by human annotators who viewed a conversation as a reference class for model goal inferences in that same conversation. We introduce GoalBench, which integrates a dataset of 4,731 human-written, validated goals across 776 conversation turns, as well as a dataset of 4,206 human goal pair similarity judgments for calibration of an automated grader, to evaluate goal inference capabilities in real human-AI interactions. We evaluated 15 models on this benchmark; while models generally perform comparably to a human baseline at producing goals contained within the human consensus, and outperform humans at recovering consensus goals, they exhibit systematic differences in the goals they produce. In particular, humans used more hedging language in their descriptions of goals and inferred more general goals relating to user motivations. Our work is a concrete step toward reliable evaluation of goal inference capabilities in realistic interactions.

Primary area:
datasets and benchmarks
Keywords:
goal inference, multi-turn, theory of mind, benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_1aNSOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the interesting problem of assessing how well AI systems can infer the goals of human users during multi-turn interactions. Towards this goal, the paper introduces a new dataset of human-written, validated goals over existing human-AI conversations, under the key insight that third party observer perception of the human conversant goals is an appropriate target of investigation. They also present experiments with a variety of models, attempting to label (or produce) the human conversant goals labeled within this dataset. These are the two main contributions of the paper.

Critical strengths

The main positive aspect, in my mind, is the initial data collection described in section 3. The procedure is well presented, and can be easily replicated or expanded upon in future studies. The detailed description of the process also leads to confidence in the quality of the data, subject to the limitations described in sections 3 and 4.

Critical weaknesses

The main weakness, in my opinion, has to do with the data collection itself, though I will note a secondary concern about the evaluation process.

The authors themselves identify some of the weaknesses of the proposed method, but it is worth listing that there are significant limitations to the scope of the data collection performed. Among these limitations, I will note the limited scale of the similarity judgments, and the lack of inter-annotator agreement (Fleiss Kappa = 0.33 for the initial set, and Cohen's Kappa = 0.44 for the held-out set). A more general weakness, in my opinion, is the overall somewhat cumbersome process of reconciliation of human judgments in the collection (i.e. the consolidation of human-written goals, described in section 4.2). This process involves developing an LLM judge to generate similarity scores for pairs of goals within a single turn in a single conversation, followed by correlation clustering over the goals, and finally a selection of single goals based on this clustering to clean up the final dataset. My concern is that this is a pretty brittle, difficult to generalize, process; it is difficult to determine how much of this process could be improved, and the details of the experiments are left out; there are no baselines produced for any of the developed components, such as the clustering algorithm or the LLM judge. How much would the rest of the results change, with only minor changes in this process? It is impossible to determine.

My other concern is about the experimental results. It appears that the only baseline used is the human baseline; however, I am wondering whether there are any automated baselines which one could also compare to in addition to this. In particular, because results claim to surpass the human baseline, but this is only due to the filtering process summarized above (and described in section 4.2 of the paper), it is difficult to assess the validity of these results in isolation.

Questions for authors

This is more of a suggestion for future work than a question, but I am wondering whether it would be better to limit the scope of the initial collection, based on these initial results, and achieve a more robust dataset which does not require all the gymnastics in section 4.2 to clean up.

3 comments
AuthorsOpenReview ↗

Thank you for the positive comments + additional information

We thank the reviewer for their positive comments about the clarity of presentation, replicability of the procedure and high quality of the dataset. Here are our responses to additional comments and questions raised:

Re: inter-annotator agreement: Thanks for raising this point - to be clear, the Cohen’s kappa value between the held-out 4th annotator judgment and the consensus judgment is a baseline, not a ceiling. It suggests that human judgments have some degree of variation, but it does not inherently mean that there is a performance ceiling for an LLM judge. It is possible that a judge can be trained to surpass human accuracy at predicting consensus judgments, e.g. because the judge has access to training examples that the human annotator does not. In fact, we show that our updated judge model achieves a Cohen’s kappa of 0.73 on the human-human test set pairs, outperforming the human baseline of 0.42. See the next section for more details about the judge.

Re: baselines for goal consolidation procedure: We have conducted additional experiments to evaluate and improve the similarity judge (varying base models, loss functions and datasets for finetuning), and report several baselines.

Keeping with our original API judge, we finetuned new judge models to predict the number of annotators who rated a goal pair as expressing the same intention (a value between 0-3), given the history of the conversation up until that point. Again, we evaluated judge performance using Cohen’s kappa between the binarized judge prediction and the majority decision of the annotators. Note that the formula for Cohen’s kappa integrates the likelihood of agreement by chance; a kappa of 0 indicates chance agreement.

Our baselines were as follows:

  • An approximate human baseline which entails collecting a 4th annotator judgment for certain goal pairs, and measuring the agreement between that held-out annotator’s decision and the majority decision of the remaining 3 annotators.
  • The original API judge model (GLM 5.3 Flash) provided with the train set examples in context (3 separate judges, for each of the different datasets: human-human judgments, human-GPT judgments, and human-Gemini judgments).

We experimented with finetuning on 3 different datasets: the train set of 1082 human-labeled examples (human-only), a synthetic dataset that was generated from the API judge models provided with the train set in context (synthetic-only), and both the human and synthetic data. The base models were Qwen 3.8 27 B and Gemma 3 27B. Results are as follows:

ModelHuman–human pairs (n = 150)Human–model pairs (n = 178)
Held-out human annotator baseline0.420.43
API judge · human–human examples0.500.42
API judge · human–Gemini examples0.530.39
API judge · human–GPT examples0.450.34
Qwen · untrained0.460.31
Gemma · untrained0.460.33
Qwen · human labels only0.730.39
Gemma · human labels only0.580.40
Qwen · synthetic labels0.530.38
Gemma · synthetic labels0.520.38
Qwen · synthetic, then human labels0.650.39
Gemma · synthetic, then human labels0.630.38

Our overall best-performing judge (Qwen finetuned on the human data) outperformed the human baseline and the API judge on the human-human goal pairs, and performed comparably to the human baseline and the API judge on the human-model goal pairs.

We will re-run the analyses with these different components and add a fuller discussion of how they contribute to the outcomes in the revision.

Re: baseline for experimental results: We appreciate the point about baselines for the benchmark itself. Because the task involves producing an open-ended output, as opposed to choosing from a fixed set of options, we are unsure how to operationalize an automated baseline. We are open to suggestions on this front. In any case, in the revision we will modify the language around the human baseline to make it clear that this is a baseline in a relatively weak sense, and should not be taken as evidence that models are “superhuman”.

Thank you for answering my questions. Please do include some of these results in the final submission of your paper. My score remains as-is but I appreciate the clarifications and believe that they will make the camera-ready submission stronger.

AuthorsOpenReview ↗

Thank you for your engagement!

Thanks for your engagement! We just wanted to close the loop on the point about the robustness of the procedure: we calculated precision/weighted recall using 2 different random seeds for the randomly-selected cluster representative. The qualitative pattern (e.g. placement of each model on the figure) stayed the same across the two seeds. We will include a figure in the appendix demonstrating this robustness.

Reviewer_3UZLOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a benchmark for evaluation open-ended goal inference in realistic conversations by collecting human annotations across 84 conversations. The papers primary novelty is to compare goal recovery to third person annotations, and focus on solving underestimation of model capabilities. The authors also collect human similarity judgements to calibrate an LLM judge for automated evaluation. Experiments show that models generally exceed human baselines in recovering goals. Results also show that models systematically under-recover abstract, hedged goals.

Critical strengths
  1. The paper is well written and clearly motivated. The use of third-person judgements instead of relying solely on first-person reports is well justified.

  2. The benchmark is a meaningful contribution with clear relevance to downstream tasks. The distinction between plausible and likely goals is also useful and makes the evaluation more nuanced.

  3. I like the rigor of the dataset collection and validation process, particularly the manual validation of goal annotations and use of human similarity judgements to calibrate the judge. Moreover, I was able to reproduce the trends of experimental results.

Critical weaknesses
  1. Annotators are instructed to track goals as active, resolved or abandoned, but the paper evaluates newly written goals at each turn. If the model correctly infers resolved/abandoned goals then the method is unfairly penalizing it. The authors do acknowledge this, but its unclear how much impact does this have on the precision.

  2. The benchmark relies on human annotation as the ground truth for goal inference, but the granularity of the goals remains underspecified. Immediate goals and underlying goals may have varying levels of same intention, which can make the goal matching process noisy. Human agreement on goal-pair similarity is also modest, and more than half of the resulting clusters contain only one annotation.

Questions for authors

Clarifying Questions:

  1. Can the authors clarify if the benchmark is intended to evaluate all currently active goals or only goals newly introduced at each turn?

  2. When two goals express the same intention, how is immediate goal vs underlying goal determined? Some more examples could be useful.

Nice to have experiments:

  1. Since models are explicitly asked to distinguish likely from plausible goals, could the authors also evaluate the accuracy of these predictions against human annotations?

  2. How much does full conversation history improve goal inference compared to using only the latest user message or the most recent exchange (bounded vs unbounded) ?

4 comments
AuthorsOpenReview ↗

Thank you for the positive comments + additional information (part 1/2)

We thank the reviewer for the positive comments; we are glad to hear that our work is “a meaningful contribution with clear relevance to downstream tasks” and appreciate the comments about the rigor and reproducibility of our data collection procedure. We respond to additional comments and questions below:

Re: active vs newly written goals: The original submission evaluated goals newly introduced at each turn. To address this question and allow for comparison of all currently active human-written goals against model goals, we finetuned a bespoke similarity judge model, experimenting with different base models, loss functions and datasets to identify a best-performing judge. The key technical challenge is that the number of pairwise comparisons for our merging process scales quadratically in the number of goals to be consolidated in a turn, making the use of an API judge infeasible from a cost perspective if we wish to extract similarity judgments for all human goal pairs on a turn. This motivates training a specific judge model.

Keeping with our original API judge, we trained new judge models to predict the number of annotators who rated a goal pair as expressing the same intention (a value between 0-3), given the history of the conversation up until that point. Again, we evaluated judge performance using Cohen’s kappa between the binarized judge prediction and the majority decision of the annotators. Note that the formula for Cohen’s kappa integrates the likelihood of agreement by chance; a kappa of 0 indicates chance agreement.

Our baselines were as follows:

  • An approximate human baseline which entails collecting a 4th annotator judgment for certain goal pairs, and measuring the agreement between that held-out annotator’s decision and the majority decision of the remaining 3 annotators.
  • The original API judge model (GLM 5.3 Flash) provided with the train set examples in context (3 separate judges, for each of the different datasets: human-human judgments, human-GPT judgments, and human-Gemini judgments).

We experimented with finetuning on 3 different datasets: the train set of 1082 human-labeled examples (human-only), a synthetic dataset that was generated from the API judge models provided with the train set in context (synthetic-only), and both the human and synthetic data. The base models were Qwen 3.8 27 B and Gemma 3 27B. Results are as follows:

ModelHuman–human pairs (n = 150)Human–model pairs (n = 178)
Held-out human annotator baseline0.420.43
API judge · human–human examples0.500.42
API judge · human–Gemini examples0.530.39
API judge · human–GPT examples0.450.34
Qwen · untrained0.460.31
Gemma · untrained0.460.33
Qwen · human labels only0.730.39
Gemma · human labels only0.580.40
Qwen · synthetic labels0.530.38
Gemma · synthetic labels0.520.38
Qwen · synthetic, then human labels0.650.39
Gemma · synthetic, then human labels0.630.38

Our overall best-performing judge (Qwen finetuned on the human data) outperformed the human baseline and the API judge on the human-human goal pairs, and performed comparably to the human baseline and the API judge on the human-model goal pairs.

We are working as quickly as possible to compute the updated precision scores with the new scalable judge (it requires a large bulk inference job, because of the quadratic scaling issue we mentioned earlier); we will report these numbers as soon as we have them.

AuthorsOpenReview ↗

Thank you for the positive comments + additional information (part 2/2)

Re: granularity of goals and similarity judgments: We intentionally gave annotators relatively limited guidelines regarding the granularity of the goals - because this is the first dataset of its kind, we wanted to cast a wide net and elicit a broad assortment of goals. The flip side is that certain goals may be idiosyncratic to individual annotators (reflected in the fact that certain goal clusters only have one associated goal). We accounted for this potential idiosyncrasy by using weighted recall as one of our key metrics, rather than just recall: this means that models are less penalized for missing a goal that only one annotator produced, compared to a goal that was produced by multiple annotators.

Regarding the similarity judgments - in the annotator onboarding procedure, we tried to orient similarity annotators to lean on the more literal side, so goals that exist at different levels of abstraction would not be marked as the same (we will include an example from the onboarding procedure in the revision).

That said, we did observe variation in the similarity judgments. We took deliberate steps to account for this by collecting multiple judgments per goal pair and scoping the judge task to predicting the majority decision, rather than a single annotator’s judgment. But it is true that these are inherently subjective judgments with some degree of variation. Our goal here is to establish a core procedure for studying these kinds of complex and nuanced inferences and we hope that others can build atop it and continually refine it.

Re: nice to have experiments: Thanks for raising these ideas. We are happy to add these experiments to the revision.

Reviewer_3UZLOpenReview ↗

Response to Authors

Thank you to the authors for their response. The rebuttal addresses my concern regarding goal granularity. The remaining results do not meaningfully affect the score.

Overall, I'll maintain my original scores and recommendation for acceptance.

AuthorsOpenReview ↗

Thank you for your engagement!

Thanks for your response! Just to close the loop on the point about the outstanding results: our preliminary results show that our improved judge recapitulated the overall ranking of models for the subset of goals that were newly written, while the ordering looks different when examining only the goals that were carried over from previous turns - in particular, the top-line OpenAI and Anthropic models are no longer on the Pareto frontier (e.g., these models are less likely to produce goals that resemble the carried-over ones produced by human annotators). We think this might reflect a recency bias. Overall, aggregating across all categories of goals, the precision is generally higher, while the recall is lower. We are further analyzing these additional results and will integrate them into the camera-ready.

Reviewer_Uw4iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper at hand introduces GoalBench, a benchmark for evaluating whether language models can infer user goals from real multi-turn conversations. The paper first motivates third-person human judgments as a practical reference for open-ended goal inference. It then collects plausible and likely goals from human annotators across 776 conversation turns, filters these annotations for quality, and clusters them into shared interpretations. Next, the authors collect human similarity judgments to calibrate a model-based grader that compares generated goals with the human goal clusters. They use this grader to evaluate 15 models through weighted recall and precision under both unbounded generation and a controlled three-goal setting. Finally, the paper examines which goals models tend to miss, finding that models recover immediate and concrete goals more reliably than underlying motivations or goals expressed with annotator hedging. The paper concludes by discussing the tradeoff between recovering more plausible goals and incorrectly attributing goals to users.

Critical strengths
  • The paper studies a genuinely important problem and turns it into a reusable evaluation task. Goal inference matters for assistants that must follow implicit, changing, and hierarchically related intentions across multiple turns. Existing synthetic and closed-set evaluations do not capture this ambiguity well. GoalBench therefore offers the ICLR community a useful dataset and evaluation setup grounded in real conversations. This supports acceptance because the benchmark could enable direct comparisons and further work on conversational goal tracking.
  • The experiments address an important confound and produce findings that go beyond a leaderboard. The bounded condition shows that recall differences partly depend on how many goals a model generates. The missed-goal analysis then identifies a consistent difference between immediate requests and broader motivations. These experiments support acceptance because they demonstrate that the benchmark separates models and can reveal interpretable behavioral differences.
Critical weaknesses
  • The paper does not establish that its evaluation target corresponds to user intent. GoalBench measures agreement with third-person observers, but several claims describe this as inferring what users actually want. From a pragmatic perspective, observer consensus may recover conventional speech acts while missing particularized implicatures, private motivations, or context unavailable in the transcript. This supports rejection because the central construct remains unvalidated. Strong model performance may mean that a model predicts crowd consensus rather than understands the user.
    • line 40: This sentence changes the target from a goal in the user’s head to an observer’s interpretation without establishing equivalence.
    • line 162: “Clear enough that the AI agent should act” defines likelihood through actionability rather than probability.
    • line 171: Requiring one likely goal per turn may force an annotation when the user only acknowledges an answer or closes the conversation.
    • line 215: Rewarding agreement with the majority may suppress reasonable minority interpretations.
    • line 436: “Might” and “possibly” express the annotator’s confidence, not necessarily uncertainty in the user’s goal.
  • The model and human reference sets may use different temporal definitions of a goal. Models are instructed to produce all active goals, while the discussion indicates that the reference contains goals newly written for the current turn. The authors themselves note that broader project goals may receive low precision for this reason. The moderate grader agreement and random cluster representative add further uncertainty. This supports rejection because the mismatch directly affects precision, model rankings, and the main conclusion about which goals models incorrectly produce.
    • line 260: Please state the criterion used to select GLM 5.3 Flash over the other candidate judges.
    • line 273: A random representative can change the score even when the underlying cluster remains fixed.
    • line 310: Please report whether model rankings remain stable under similarity thresholds other than two.
    • line 394: “Better than individual crowdworkers” should be narrowed to “better at predicting the three-annotator consensus.”
    • line 448: This line states that the clusters are based on goals newly written for the turn.
    • line 709: The model prompt instead requests every goal that remains active at that point.
    • line 477: The recommendation to generate more goals under uncertainty is not directly tested.
Questions for authors
  • At each turn, does the reference set contain every human goal marked active or only goals newly written on that turn? If it contains only new goals, can the authors recompute precision using all active goals?
  • Did the authors compare the third-person annotations with the first-person information available in ThoughtTrace? Evidence that these sources agree would materially strengthen the construct validity of GoalBench.
  • Were the quality classifier and similarity judge validated with entire conversations and annotators held out? If not, how much do their results change under conversation-level splits?
  • Do model rankings remain stable when the similarity threshold, judge model, and selected cluster representative are changed? This would determine whether the reported ordering reflects model behavior or grader choices.
3 comments
AuthorsOpenReview ↗

Thank you for the positive comments + additional information (part 1/2)

We thank the reviewer for their positive comments; we appreciate the comment that our work provides a “useful dataset and evaluation setup grounded in real conversations”. We address further comments and questions below:

Re: the evaluation target: Our objective is to measure model agreement with third-person observers’ assessment of a user’s goal. This operationalization of goals is what matters for assistants writ large (whether the assistant is a person or a model): we want assistants to act on intent in ways that a human would, motivating the use of a third-person human goal inference as the reference class.

We are not claiming that the 3rd-person consensus is necessarily the same as the user’s true goal. Instead, our claim is that 3rd-person judgments are a more reasonable and operationalizable reference class for model inferences about user goals, compared to 1st-person reports of goals. This for the aforementioned reason, and also because 1) in the first-person setting, individual users have access to their own mental state in ways that models clearly don’t, which could underestimate model performance (e.g. because some goals are simply unknowable), and 2) 1st-person self reports may not reflect users’ true intentions, as people may lack awareness of their goals. For instance, a person who shares personal relationship details with AI may not be consciously aware they are using AI as a source of emotional validation and therefore may not report that as a goal, whereas a third-person observer of the transcript would likely come to that conclusion. We will revise the paper to make the motivation for 3rd-person observations more clear, and also distinguish between 3rd-person and 1st-person judgments.

Re: active vs newly written goals: The original submission evaluated goals newly introduced at each turn. To address this question and allow for comparison of all currently active human-written goals against model goals, we finetuned a bespoke similarity judge model, experimenting with different base models, loss functions and datasets to identify a best-performing judge. The key technical challenge is that the number of pairwise comparisons for our merging process scales quadratically in the number of goals to be consolidated in a turn, making the use of an API judge infeasible from a cost perspective if we wish to extract similarity judgments for all human goal pairs on a turn. This motivates training a specific judge model.

Keeping with our original API judge, we trained new judge models to predict the number of annotators who rated a goal pair as expressing the same intention (a value between 0-3), given the history of the conversation up until that point. Again, we evaluated judge performance using Cohen’s kappa between the binarized judge prediction and the majority decision of the annotators. Note that the formula for Cohen’s kappa integrates the likelihood of agreement by chance; a kappa of 0 indicates chance agreement.

Our baselines were as follows:

  • An approximate human baseline which entails collecting a 4th annotator judgment for certain goal pairs, and measuring the agreement between that held-out annotator’s decision and the majority decision of the remaining 3 annotators.
  • The original API judge model (GLM 5.3 Flash) provided with the train set examples in context (3 separate judges, for each of the different datasets: human-human judgments, human-GPT judgments, and human-Gemini judgments).

We experimented with finetuning on 3 different datasets: the train set of 1082 human-labeled examples (human-only), a synthetic dataset that was generated from the API judge models provided with the train set in context (synthetic-only), and both the human and synthetic data. The base models were Qwen 3.8 27 B and Gemma 3 27B. Results are as follows:

ModelHuman–human pairs (n = 150)Human–model pairs (n = 178)
Held-out human annotator baseline0.420.43
API judge · human–human examples0.500.42
API judge · human–Gemini examples0.530.39
API judge · human–GPT examples0.450.34
Qwen · untrained0.460.31
Gemma · untrained0.460.33
Qwen · human labels only0.730.39
Gemma · human labels only0.580.40
Qwen · synthetic labels0.530.38
Gemma · synthetic labels0.520.38
Qwen · synthetic, then human labels0.650.39
Gemma · synthetic, then human labels0.630.38

Our overall best-performing judge (Qwen finetuned on the human data) outperformed the human baseline and the API judge on the human-human goal pairs, and performed comparably to the human baseline and the API judge on the human-model goal pairs.

We are working as quickly as possible to compute the updated precision scores with the new scalable judge (it requires a large bulk inference job, because of the quadratic scaling issue we mentioned earlier); we will report these numbers as soon as we have them.

AuthorsOpenReview ↗

Thank you for the positive comments + additional information (part 2/2)

Re: third person annotations vs first person info from ThoughtTrace: We did examine the self-report annotations in the ThoughtTrace dataset; however, only 35% of the messages in our subset have an associated rationale, and many convey generic information (e.g. “good response”), rather than explicit information about user goals. This in part motivated our focus on third-person annotations as a more operationalizable reference class for model predictions.

Re: classifier and judge validation: Both the quality classifier and the similarity judge were validated with entire conversations held out (see the updated similarity judge metrics). For the quality classifier, the first author of the paper annotated goals associated with 15 different conversations for quality. We then conducted k-fold validation, where goals from 5 conversations were held out entirely and the remaining 10 conversations’ worth of goals were used as in-context examples for the classifier. The metrics reported reflect the mean over 3 folds.

For the similarity judge, we held out similarity judgments from 3 conversations from the human annotations. We did not intentionally sample our test examples to hold out individual annotators, although in the test set for the human-model annotations, there were annotators who happened to be fully unseen.

Reviewer_Uw4iOpenReview ↗

Response to author replies

Thank you very much for your answers to my questions and checking the firs person information from the dataset. I believe that distinction creates dynamic differences, but for the context of this paper, this is acceptable. I will not be changing my review score as it is already on the higher end, but I believe this paper should be part of the conference, as it provides a much needed focus on multi-turn interactions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.