- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TurnFlow, a benchmark and evaluation framework for dynamic multi-turn collaboration. From 250,100 real sessions, the authors build 400 tasks covering five interaction-path types (STI, MTI, REC, CCJ, GPC), each paired with one of 16 user communication styles. A path planner and a user simulator generate dialogues with the tested model, a recorder logs turn-level evidence, and three LLM judges score the sealed trajectories on four metrics: TCR, TCE, SSTC, and BGV.
Critical strengths
- The authors focus on evaluating LLM on complex multi-turn tasks. The benchmark is carefuily designed and new metrics are introduced to the task. The evaluation suite can be useful to the community.
- The experiments bring some novel findings. The results show that BGV is rare and does not consistently track task completion. SSTC also reveals performance drops at specific interaction stages that final-success metrics miss. The worked scoring example makes the evaluation easier to trace.
Critical weaknesses
- In experimental results, the human study reports 55.1% agreement after excluding automatic ties, without inter-annotator agreement or single-metric baselines. It therefore does not establish whether combining the four metrics improves on TCR or SSTC alone. The shared simulator–judge model and unspecified Recorder model also raise concerns about robustness to auxiliary-model choices.
- TCR is higher for question-style conversations in all seven models, but six of the seven confidence intervals include zero. Different styles are tested on different tasks, while later stages also involve more turns and longer context. These comparisons therefore provide limited support for the claim that models struggle to use earlier context.
Questions for authors
- In App. D.3, why is the outsider subgoal eligible at turn 1? Is eligibility determined only from information visible to the tested model?
- Per model, what are the NA rates and terminal-status counts (satisfied, abandon, truncated)? TCR excludes indeterminate cases, while TCE additionally includes only successfully completed tasks.
- The code defines a sixth path type (MEM), and data/README.md says MEM dialogues were excluded. Are any of them part of the reported results?