- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates how choices made when structuring multi-turn conversations impact the benchmark scores and conclusions of Large Language Model (LLM) evaluations. They test various ways of “designing” conversations on the lost-in-conversation dataset (Laban et al., 2026) and cross 5-open weight models. They find that changing the conversation design (the way turns are ordered or blocked together) changes how the models score on the benchmark.
Critical strengths
-
novel insight on why models perform badly during multi-turn conversations. haven’t seen any other works evaluate models on the ordering of the information released. in reality, however, i question how “conversation design” could be used in the real world: it would be difficult to get users to run their conversations in a particular order.
-
the study design is comprehensive (authors tested on 5 open weight models) and tested out several “design” choices (front-loading, ordering, granularity etc.)
Critical weaknesses
-
my major complaint is that all experiments were done on the synthetic dataset of Laban et al., (2026), a dataset that has been known by folks in this space to be non-representative of real-world conversations (each conversation is forced into shards).
-
the Laban paper also investigated various configurations of how to group lines in a multi-turn conversation. the authors should present a stronger case for how their methods/study differ substantially from earlier methods tested by Laban et al and Huang et al.
Questions for authors
who is the main audience of this study? is it users (to learn how to optimally structure their conversations?) or engineers (to learn how to better structure harnesses?) for instance, if model providers were to report a “design variance” score, then how would users find this useful or know which model to use?
3 comments
Thanks to the authors for revising the manuscript to clarify the audience (benchmark developers, evaluators) and scope, and for clarifying the limitations of the synthetic dataset. My primary concern with this work is the practical relevance of studying the design of conversations. The ordering of a multi-turn conversation is dictated by the user and is not something that model providers can control, so a benchmark built around reordering conversation turns may be of limited practical relevance. Added to that, the findings are limited to a synthetic dataset. For these reasons, I am keeping to my original score.
The practical question we study comes up when an evaluator constructs conversations from complete tasks, as existing sharded benchmarks do. The evaluator must choose how information is distributed across turns. Our results show that this choice can materially affect the conclusions drawn from that benchmark. An evaluator using only one design may overstate or understate a mitigation’s benefit relative to the average over the studied family. Measuring variation across designs helps determine how dependent a reported improvement is on the chosen evaluation protocol. It is not intended as a standalone score for users selecting a model.
We agree that these controlled experiments do not establish the magnitude of the effect in naturally occurring conversations. Our contribution is an audit of an existing synthetic evaluation practice. It shows why conclusions from that practice should be qualified by the conversation construction used.
Thank you for your review. We appreciate the opportunity to clarify the intended audience, the dataset limitations, and how our work differs from prior studies.
Our primary audience is benchmark developers and evaluators, rather than end users. We agree that asking users to structure conversations in a particular way would be impractical. Our point is that when evaluators choose a single conversation structure, their results may reflect that choice rather than general multi-turn performance. We therefore recommend specifying the design used and reporting scores across multiple designs when making broader claims. Design variation also helps interpret model comparisons: in our experiments, large performance gaps remain stable across designs, while smaller gaps can reverse in point estimates, although these reversals were not established on held-out instances (Section 5.5).
We agree that the synthetic nature of the dataset limits generalizability and will make this clearer in the paper. However, the controlled structure is also important for isolating the effect we study. Our experiments hold the requirements, task instances, and checkers fixed while varying how requirements are distributed across turns. This allows us to attribute differences to conversation design rather than changes in content. We acknowledge that converting the original tasks into requirement units can itself affect performance, and we report this separately in Table 10. We considered IFEval as an alternative, but its limited number of verifiable constraints would not support the turn counts studied here. More importantly, because this benchmark is already used to evaluate multi-turn performance and mitigation methods, understanding its sensitivity to conversation design is directly relevant. We will identify evaluation on naturally occurring conversations as an important direction for future work.
Our contribution also differs from Laban et al. and Huang et al. Laban et al. investigate performance degradation in multi-turn conversations and examine shard granularity, whereas we investigate how evaluation conclusions change across a systematic set of conversation designs. We vary four factors across 24 designs (20 for database) and evaluate the stability of scores, model rankings, and mitigation gains using held-out comparisons and variance estimates. Our matched one-message controls additionally separate the effect of requirement arrangement from the cost of multi-turn interaction. Unlike Laban et al., who test two mitigations under the default design, we examine four across designs with a placebo control. Huang et al. propose removing earlier assistant replies; we evaluate this method and find that its effectiveness depends on conversation design. On tool selection, its gain is +0.540 at the reference design versus +0.353 on average. Its performance also closely matches the corresponding one-message control (Table 6), suggesting that it removes the interaction penalty without addressing the effect of requirement arrangement. We will make these distinctions more explicit in Sections 1 and 2.
Thank you for raising these points. We will revise the manuscript to clarify the scope of our claims, the limitations of the dataset, and the contribution relative to previous work.