acceptodds
Under review as a conference paper at ICLR 2027

TurnFlow: Evaluating LLMs Beyond Task Completion in Dynamic Multi-Turn Interactions

Abstract

Large language models (LLMs) are increasingly used for complex collaborative tasks that require users to add constraints and revise plans over multiple turns. Model responses, in turn, shape subsequent user behavior and task progression. These dependencies call for evaluation of both task outcomes and the interaction process. However, predefined user inputs in static evaluations cannot reflect reactions to model responses, while an emphasis on final success in dynamic evaluations can obscure differences in completion efficiency, turn-level progress, and the value of proactive guidance. To address this gap, we introduce TurnFlow, a benchmark for evaluating LLMs on complex multi-turn tasks. Derived from 250,100 real-world conversation sessions, TurnFlow comprises 400 tasks covering single-goal and multi-goal settings, short and long conversations, and scenarios requiring backtracking and revision. From the same user population, we derive 16 communication style configurations that characterize how users formulate initial requests and follow-up utterances. To evaluate models on these tasks under different communication styles, we develop a dynamic multi-turn evaluation framework comprising a path planner, a user simulator, a trajectory recorder, and an evaluator. The planner releases stage-specific task information and behavioral requirements as the conversation progresses. The simulator generates user reactions based on the conversation history, user-side task information, and communication style, allowing subsequent interactions to adapt to model behavior. The recorder tracks model responses, task progress, and user feedback. Using this evidence, the evaluator assesses model performance through four trajectory-based metrics: Task Completion Rate (TCR), Task Completion Efficiency (TCE), Single-Step Task Contribution (SSTC), and Beyond-Goal Value (BGV). Together, these metrics measure task completion, completion efficiency, turn-level contributions, and value beyond prior user expectations. Our results show that models can complete tasks without consistently advancing them effectively throughout the interaction, with limitations in proactively guiding users and providing value beyond their existing needs.

Primary area:
datasets and benchmarks
Keywords:
LLMs, Dynamic multi-turn conversation, Evaluation, Benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AFisOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TurnFlow, a benchmark and evaluation framework for dynamic multi-turn collaboration. From 250,100 real sessions, the authors build 400 tasks covering five interaction-path types (STI, MTI, REC, CCJ, GPC), each paired with one of 16 user communication styles. A path planner and a user simulator generate dialogues with the tested model, a recorder logs turn-level evidence, and three LLM judges score the sealed trajectories on four metrics: TCR, TCE, SSTC, and BGV.

Critical strengths
  • The authors focus on evaluating LLM on complex multi-turn tasks. The benchmark is carefuily designed and new metrics are introduced to the task. The evaluation suite can be useful to the community.
  • The experiments bring some novel findings. The results show that BGV is rare and does not consistently track task completion. SSTC also reveals performance drops at specific interaction stages that final-success metrics miss. The worked scoring example makes the evaluation easier to trace.
Critical weaknesses
  • In experimental results, the human study reports 55.1% agreement after excluding automatic ties, without inter-annotator agreement or single-metric baselines. It therefore does not establish whether combining the four metrics improves on TCR or SSTC alone. The shared simulator–judge model and unspecified Recorder model also raise concerns about robustness to auxiliary-model choices.
  • TCR is higher for question-style conversations in all seven models, but six of the seven confidence intervals include zero. Different styles are tested on different tasks, while later stages also involve more turns and longer context. These comparisons therefore provide limited support for the claim that models struggle to use earlier context.
Questions for authors
  1. In App. D.3, why is the outsider subgoal eligible at turn 1? Is eligibility determined only from information visible to the tested model?
  2. Per model, what are the NA rates and terminal-status counts (satisfied, abandon, truncated)? TCR excludes indeterminate cases, while TCE additionally includes only successfully completed tasks.
  3. The code defines a sixth path type (MEM), and data/README.md says MEM dialogues were excluded. Are any of them part of the reported results?
Reviewer_c4gwOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on the important problem of evaluating multi-turn conversations with LLMs in a knowledge task completion setting (I emphasize "knowledge" tasks as to differentiate tasks where an LLM based system is expected to act in the world, i.e. an agentic system; this paper does not appear to focus on such tasks). The paper introduces a new benchmark corpus, TurnFlow, and an associated framework for measuring conversation quality using four metrics which they introduce as well. Additionally, the paper performs a benchmark of seven distinct LLMs using the corpus and associated framework. These are the main contributions of the paper.

Critical strengths

I did enjoy that the paper is quite easy to read and follow. I did not notice any typos and the notation seems to be quite significant.

Critical weaknesses

I have a number of concerns, chiefly related to scope, coverage of prior work, and evaluation methodology.

I will address scope and prior work first. In terms of scope, my main concern is that the paper is quite broad, in that it attempts to introduce a novel task (the benchmark dataset), an evaluation framework, as well as a set of metrics. This is a very ambitious set of goals in a very crowded space in recent years, but also one with deep body of work from the past thirty years when extending beyond LLMs and into general conversational systems. Even just focusing on LLM evaluations, I would have expected a discussion of what this paper adds relative to what has been studied; specifically, what is their perceived contribution of novelty. However, the authors do not even attempt to position their contributions in this way at all. The related works section is very short, only two paragraphs, and is woefully incomplete; there are literally hundreds of papers on LLM system evaluations out there, out of which the authors cite only four. On user simulation, which is perhaps slightly less well studied, there has nevertheless been a lot of recent work assessing which qualities user simulators should possess in order to serve as good proxies for human users, for example https://aclanthology.org/2026.acl-long.2192/ and https://aclanthology.org/2025.emnlp-main.1786/. Expanding to the broader question of designing user simulators, plenty of other papers aare available, such as https://dl.acm.org/doi/abs/10.1145/3701716.3715258 and https://aclanthology.org/2024.scichat-1.3/ and https://arxiv.org/abs/2605.09808. On frameworks for evaluation, there are also plenty of choices, including https://arxiv.org/abs/2609.38043 and https://dl.acm.org/doi/abs/10.1145/3837857 to name only a couple. There are also plenty of tasks available. So it is very disappointing to see such a sparse treatment of recent prior work.

The problem is exacerbated by the fact that the field of conversational system evaluation is not new; I would also expect any treatment of multiturn conversation evaluation to include references to classical studies and frameworks, such as Paradise, and a more general discussion of user-centric evaluation of conversational systems. None of the metrics they propose are necessarily new either, but the names they use suggest otherwise since they do not map cleanly onto other work performed in the past; for example, Single-Step Task Contribution is a combination of semantic frame matching together with a specific set of dialog acts being represented in the LLM response. Which leads to another observation; the authors appear to be unfamiliar with the rich body of literature discussing dialog acts (and, related, speech acts), since they come up with their own nomenclature in Table 1, without any justification as to why they cannot use existing schemas. All of these are deep problems with the paper in its current form.

Finally, on evaluation methodology. As mentioned above, there are plenty of frameworks for evaluating LLM conversations; the authors, however, do not deem necessary to use any of them as baselines. This makes the results presented entirely meaningless, as there is no way of determining whether their proposed framework is in fact better at achieving their purpose than other options out there (such as some of the ones listed above). Furthermore, their dataset is very small (only 400 samples) and the distribution is skewed towards one type of conversation for no clear reason. There is also no notion of statistical significance anywhere; even accepting that their methodology is sound, are the differences between the seven LLMs used in the study significant anywhere, or not? How should the results be interpreted?

Questions for authors

These are primarily suggestions for future work. I would suggest:

  • an in-depth study of what is already available in the literature
  • a selection of 2-3 research questions being answered in the publication
  • an experimental design which can answer these questions, including clear baselines for frameworks and potentially for datasets
  • a clear evaluation of the proposed methods against baselines
  • a detailed discussion interpreting the results, including statistical significance.
Reviewer_4RwFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TurnFlow, a benchmark for evaluating LLMs in dynamic multi-turn interactions. The authors argue that existing evaluations focus too much on final task completion and often overlook interaction efficiency, turn-level contributions, and the additional value a model brings to users.

TurnFlow contains 400 tasks derived from real-world conversations, covering different interaction paths and 16 communication styles. The evaluation framework consists of a Path Planner, User Simulator, Trajectory Recorder, and Evaluator, allowing user responses to adapt to model behavior.

The authors also introduce four metrics: Task Completion Rate (TCR), Task Completion Efficiency (TCE), Single-Step Task Contribution (SSTC), and Beyond-Goal Value (BGV). Experiments on seven LLMs reveal differences in task completion, interaction efficiency, and proactive assistance.

Critical strengths
  1. The paper addresses an important limitation of existing multi-turn benchmarks: final task completion alone does not fully capture how well an LLM collaborates with users. Separating task completion, interaction efficiency, and proactive assistance is a useful direction for evaluating multi-turn capabilities.

  2. The authors build tasks from real-world conversations and introduce different interaction paths and communication styles. The framework also separates dynamic interaction generation from final evaluation. Experiments across seven models, along with human evaluation, provide a useful starting point for studying multi-turn collaboration.

Critical weaknesses

1.Limited validation of the proposed evaluation metrics. The paper introduces four metrics for multi-turn collaboration, but the human alignment results in Section 5.3 are not particularly convincing. Among model pairs receiving a non-tie judgment, agreement with the majority of three human annotators is only 56.4%. This raises questions about how well the metrics capture human judgments of interaction quality.

The paper also lacks sufficient analysis of inter-annotator agreement and comparisons with simpler evaluation approaches, such as direct LLM-as-a-Judge comparisons. Without these results, it is difficult to tell whether the proposed metrics offer a more reliable evaluation. This is my main concern with the paper.

2.The reliability of simulated user feedback is not sufficiently validated. TurnFlow relies on a User Simulator to generate dynamic feedback, and BGV further depends on whether the simulated user accepts or supports model suggestions. However, simulated acceptance does not necessarily reflect what real users would find useful.

The BGV scores in Table 2 are also quite low, ranging from 2.2 to 4.4 out of 100. The paper provides limited analysis of the metric's ability to distinguish models or whether its results remain stable with different user simulators. More evidence is needed to establish BGV as a reliable measure of additional value to users.

Questions for authors

1.Section 5.3 reports 56.4% agreement with the human majority. Could the authors report inter-annotator agreement and compare the proposed metrics with a direct LLM-as-a-Judge baseline? This would help clarify how much value the four metrics add.

2.Have the authors tested whether the evaluation results remain stable with different user simulators? For BGV in particular, is there any evidence that simulated user acceptance aligns with real user preferences? This seems important for validating the metric.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.