acceptodds
Under review as a conference paper at ICLR 2027

THE TURN PARTITION IS A FREE PARAMETER IN MULTI-TURN LLM EVALUATION

Abstract

Large language models often score lower when a complete request is split across several conversation turns. Measuring this gap requires choosing how to turn the request into a conversation. We study how that choice affects the conclusions of a multi-turn evaluation. Using four tasks from the sharded benchmark of Laban et al. (2026), we evaluate five open-weight models across 24 conversation designs. These designs vary the number of turns, requirement order, the amount of information provided initially, and whether an earlier requirement is repeated, while preserving the final requirements and evaluation checker. We observe that conversation design substantially changes measured performance: the best and worst designs differ by a median of 0.123 on a 0-1 scale. Gaps between models can be small relative to their variation across designs, and the gains from four mitigation methods vary across designs, sometimes changing sign. When the same content is given in a single message, performance still varies across designs, indicating that design sensitivity is not exclusive to interactive conversations. These results show that conclusions about multi-turn performance and mitigation effectiveness are conditional on the conversation design used to evaluate them. We recommend evaluating across multiple explicitly specified designs and reporting variation alongside average performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Llm, evaluation, multi turn partition, machine learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2QV1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates how choices made when structuring multi-turn conversations impact the benchmark scores and conclusions of Large Language Model (LLM) evaluations. They test various ways of “designing” conversations on the lost-in-conversation dataset (Laban et al., 2026) and cross 5-open weight models. They find that changing the conversation design (the way turns are ordered or blocked together) changes how the models score on the benchmark.

Critical strengths
  1. novel insight on why models perform badly during multi-turn conversations. haven’t seen any other works evaluate models on the ordering of the information released. in reality, however, i question how “conversation design” could be used in the real world: it would be difficult to get users to run their conversations in a particular order.

  2. the study design is comprehensive (authors tested on 5 open weight models) and tested out several “design” choices (front-loading, ordering, granularity etc.)

Critical weaknesses
  1. my major complaint is that all experiments were done on the synthetic dataset of Laban et al., (2026), a dataset that has been known by folks in this space to be non-representative of real-world conversations (each conversation is forced into shards).

  2. the Laban paper also investigated various configurations of how to group lines in a multi-turn conversation. the authors should present a stronger case for how their methods/study differ substantially from earlier methods tested by Laban et al and Huang et al.

Questions for authors

who is the main audience of this study? is it users (to learn how to optimally structure their conversations?) or engineers (to learn how to better structure harnesses?) for instance, if model providers were to report a “design variance” score, then how would users find this useful or know which model to use?

3 comments

Thank you for your review. We appreciate the opportunity to clarify the intended audience, the dataset limitations, and how our work differs from prior studies.

Our primary audience is benchmark developers and evaluators, rather than end users. We agree that asking users to structure conversations in a particular way would be impractical. Our point is that when evaluators choose a single conversation structure, their results may reflect that choice rather than general multi-turn performance. We therefore recommend specifying the design used and reporting scores across multiple designs when making broader claims. Design variation also helps interpret model comparisons: in our experiments, large performance gaps remain stable across designs, while smaller gaps can reverse in point estimates, although these reversals were not established on held-out instances (Section 5.5).

We agree that the synthetic nature of the dataset limits generalizability and will make this clearer in the paper. However, the controlled structure is also important for isolating the effect we study. Our experiments hold the requirements, task instances, and checkers fixed while varying how requirements are distributed across turns. This allows us to attribute differences to conversation design rather than changes in content. We acknowledge that converting the original tasks into requirement units can itself affect performance, and we report this separately in Table 10. We considered IFEval as an alternative, but its limited number of verifiable constraints would not support the turn counts studied here. More importantly, because this benchmark is already used to evaluate multi-turn performance and mitigation methods, understanding its sensitivity to conversation design is directly relevant. We will identify evaluation on naturally occurring conversations as an important direction for future work.

Our contribution also differs from Laban et al. and Huang et al. Laban et al. investigate performance degradation in multi-turn conversations and examine shard granularity, whereas we investigate how evaluation conclusions change across a systematic set of conversation designs. We vary four factors across 24 designs (20 for database) and evaluate the stability of scores, model rankings, and mitigation gains using held-out comparisons and variance estimates. Our matched one-message controls additionally separate the effect of requirement arrangement from the cost of multi-turn interaction. Unlike Laban et al., who test two mitigations under the default design, we examine four across designs with a placebo control. Huang et al. propose removing earlier assistant replies; we evaluate this method and find that its effectiveness depends on conversation design. On tool selection, its gain is +0.540 at the reference design versus +0.353 on average. Its performance also closely matches the corresponding one-message control (Table 6), suggesting that it removes the interaction penalty without addressing the effect of requirement arrangement. We will make these distinctions more explicit in Sections 1 and 2.

Thank you for raising these points. We will revise the manuscript to clarify the scope of our claims, the limitations of the dataset, and the contribution relative to previous work.

Thanks to the authors for revising the manuscript to clarify the audience (benchmark developers, evaluators) and scope, and for clarifying the limitations of the synthetic dataset. My primary concern with this work is the practical relevance of studying the design of conversations. The ordering of a multi-turn conversation is dictated by the user and is not something that model providers can control, so a benchmark built around reordering conversation turns may be of limited practical relevance. Added to that, the findings are limited to a synthetic dataset. For these reasons, I am keeping to my original score.

The practical question we study comes up when an evaluator constructs conversations from complete tasks, as existing sharded benchmarks do. The evaluator must choose how information is distributed across turns. Our results show that this choice can materially affect the conclusions drawn from that benchmark. An evaluator using only one design may overstate or understate a mitigation’s benefit relative to the average over the studied family. Measuring variation across designs helps determine how dependent a reported improvement is on the chosen evaluation protocol. It is not intended as a standalone score for users selecting a model.

We agree that these controlled experiments do not establish the magnitude of the effect in naturally occurring conversations. Our contribution is an audit of an existing synthetic evaluation practice. It shows why conclusions from that practice should be qualified by the conversation construction used.

Reviewer_CigFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper studies how multi-turn evaluation changes when the same requirements are delivered through different conversation designs. It varies turn count, requirement order, initial information, and restatement across 24 designs, with 20 designs for the database task, using four tasks and five open-weight models. It compares original full prompts, matched concatenated prompts, and interactive conversations, and tests four mitigation methods on Qwen2.5-7B.

Across the 11 settings retained in the main analysis, the median bias-corrected standard deviation across designs is 0.067, and the median held-out difference between selected designs is 0.123. The results support sensitivity in scores and mitigation gains, while model-ranking reversals are not established on new instances. The paper recommends reporting averages and variation over explicitly specified and weighted conversation designs.

Critical strengths
  1. The main result has support beyond selected best and worst scores. Table 21 reports positive held-out design contrasts in 10 of 11 retained settings, and Appendix O checks that the main contrasts persist after regeneration and randomized submission order. The Gemma replication also finds design sensitivity on three tasks (Table 20), strengthening the evidence beyond the Qwen family.
  2. The matched control makes the findings useful for evaluation practice. CONCAT-BLOCKS preserves requirement order and grouping, allowing the study to measure the additional penalty associated with delivering those blocks across turns (Section 5.7). The direct comparison with user-turn-only is especially helpful. On Qwen2.5-7B math, its score is 0.666 against 0.660 for the matched control, clarifying that its improvement largely recovers performance available without earlier assistant replies (Table 6).
Critical weaknesses
  1. The ordering procedure is not consistently specified. Section 3.1 says the request is the only non-constraint unit and that order permutes only the requirements following it, which would make constraint-first and canonical identical. Appendix P instead says they coincide only when the opening block contains one unit, and Table 23 retains both at otherwise matching settings. This ambiguity matters because order is reported as the dominant factor on math and database, so the precise manipulation needs to be clear.

  2. The database mitigation evidence still uses an extractor with demonstrated errors. Deterministic rescoring raises database means from 0.289 to 0.391 at 7B and from 0.309 to 0.523 at 14B, while removing the database ranking reversals (Table 16; Section 5.5). No corresponding corrected mitigation results are reported, including for the recap example of −0.104 at the reference design versus +0.015 averaged over designs (Section 5.6). The baseline sensitivity result survives correction, but this particular example of changing mitigation effectiveness could still reflect differential extraction errors.

Questions for authors
  1. Does request-first mean that the request is the first requirement within turn 1, or that it appears somewhere in turn 1? Please provide a rendered canonical/constraint-first pair for the same instance with a multi-requirement opening block, and reconcile Section 3.1 with Appendix P.
  2. How do the database mitigation gains and matched CONCAT-BLOCKS versus SHARDED comparisons change when every arm uses the deterministic SQL extractor? In particular, does the recap reference-versus-average contrast survive, and which database mitigation gains still vary significantly across designs?
5 comments

Thank you for your review. Both points are addressed below.

On the ordering procedure, request-first means the request appears somewhere in turn 1, not necessarily first. Appendix P is correct, and we will remove the contradictory sentence in Section 3.1. Canonical places the request before the initial constraints, while constraint-first places it after those same constraints within turn 1. The subsequent turns remain identical. With a one-unit opening block, the two coincide and the duplicate is dropped. Reversed and random also keep the request in turn 1. We will add a rendered canonical/constraint-first pair to Appendix P. In matched comparisons across the 11 main-grid settings, constraint-first differs from canonical by a median of only −0.010, compared with −0.091 for random and −0.113 for reversed. This suggests that the ordering effect is driven primarily by constraint order rather than request position.

On database extraction, we agree that the mitigation results needed verification because the submitted analysis used different extractors across arms. We re-evaluated all database conditions using both extractors and a common execution checker. Baseline design sensitivity remains significant under both (all p < 0.001), while the matched CONCAT-BLOCKS versus SHARDED penalty approximately doubles under deterministic extraction, reaching +0.145 at both 7B and 14B. These findings therefore remain robust to the extraction correction.

The database recap interaction, however, does not replicate, and we withdraw it. Its reference-versus-average difference is −0.069 [−0.165, +0.033] under deterministic extraction and +0.011 [−0.086, +0.110] under LLM extraction, neither significant. Snowballing is the only database mitigation showing significant design-dependent gains (p = 0.001, BH-corrected). The overall count remains 10 of 16 task–method pairs, now comprising 9 of 12 across math, tool selection, and code, and 1 of 4 on database. User-turn-only's average database gain also increases from +0.094 to +0.148.

We will remove the database recap example from Section 5.6, retain the tool-selection example, report the corrected results in two new appendix tables, and mark the affected database rows of Tables 3 and 5 as superseded.

Thank you for catching these points.

We wanted to follow up on our response posted on November 12. As the discussion closes on November 18 (AoE), please let us know if any concerns remain or if further clarification would be helpful so we can respond before the deadline.

If we have adressed all your concerns, please update your rating, thank you for your time and feedback.

We have uploaded a revised PDF incorporating the changes described in our response, with the main revisions highlighted in blue. As the discussion closes on November 18 (AoE), please let us know if you have any further questions or if any points still need clarification or revision.

Reviewer_ze9LOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors studies the Lost-in-conversation phenomenon in multi-turn LLM evaluation, specifically, by evaluating conversations according to some custom axes (#turns, information upfront, order, restatement) and investigating the effect on the score, model comparisons, and mitigation gains. The authors additionally experimented with single-message control with the same arrangement, in an effort to study whether the sensitivity comes from the interaction. While shard granularity (ie #turns) and order have been somewhat explored in Lost-in-Conversation, the main contribution and distinction in this paper is a more detailed study on sharding design, which they call "conversation design."

The authors report that the task scores swing substantially using different conversation designs, with no design choices filling the gap to the fully-specified setting. Through analyses across experimental settings, the authors recommend to report results across multiple conversation designs for these evaluations.

That said, I have a few concerns about how the experiments are conducted. First, is the variance accounted for enough to support the claims? Each design is simulated with only 3 seeds if I'm not mistaken, while the very lost-in-conversation work reports that the variance across simulations is large (even with temp=0), so 3 simulations might not be sufficient to dismiss the possibility of simulation noise, though I do acknowledge that the rerun results in the appendix partly address this for a few settings. Small simulation trials also disables measuring the aptitude & unreliability (10 to 90-percentile score gaps). A key design decision is the number of turns, by merging shards into fixed turns, which additionally cause variance and introduces fragility of those shards re: whether they convey the same semantics or not. To be more thorough, the authors would ideally sample N T=2/4/6-versions of merged shards, then average the observations over K simulations for each variant, based on N*K rollouts.

Next, it's unclear if we can explain the findings with a universal trend besides stating the sensitivity to proposed designs. It seems that which design component matters differs with different tasks and models, and the model ranking changes also have no consistencies.

Another minor observation is that the shard order dependency is also there in the Concat setting, suggesting that it's more of a order-sensitive prompt issue.

Critical strengths
  • A more detailed analysis on how sharding designs affect the evaluation.
  • Reasonable experimental results.
Critical weaknesses
  • Limited novelty over the prior work.
  • Concern on simulation scale being small, risking the findings's validity.
  • No (presumably large) frontier models are tested. We might see totally different sensitivity horizon for larger models.
Questions for authors

Please see the paper summary.

5 comments

Thank you for your review. We have conducted additional experiments to address the concerns about simulation noise, merging variability, universal trends, and model scale.

On simulation noise, we reran the grid with ten seeds per cell on Qwen2.5-7B for math and tool selection. The design means closely reproduce the original results (r = 0.96 and 0.99), and the design SDs differ by less than 0.005. We also calculated aptitude and unreliability using the 90th and 10th percentile scores, which will be reported in the appendix. These results indicate that simulation noise does not explain the observed design sensitivity.

We additionally implemented the suggested N × K protocol, sampling up to eight random merges per instance and running eight simulations of each, for 13,408 conversations. Averaged across merges and simulations, the two-turn advantage over six turns remains +0.17 on math [0.10, 0.25] and +0.18 on tool selection [0.08, 0.29]. Importantly, the merge boundaries themselves also affect performance: at fixed turn count and order, they explain 7–22% of per-conversation variance on math and 38–81% on tool selection. Thus, the turn-count effect persists when merging variability is averaged out, while the placement of boundaries is itself another source of evaluation sensitivity.

On general patterns, we find consistent directional effects across most settings. Reversed order has a negative coefficient in all 11 main-grid settings, random order in 10 of 11, four turns in 10 of 11, and six turns in 9 of 9. Models also tend to agree on which designs are difficult, with median rank correlations of 0.76–0.83 across designs, including across model families. These results suggest that reversing constraint order and spreading requirements across more turns generally reduce performance, although the magnitude varies by task and model. We will add this analysis to Appendix R and summarize it in Section 5.3. We do not claim a consistent pattern of model-ranking reversals, as none was established on held-out instances.

We agree that order sensitivity is largely a prompt-arrangement effect. In the matched one-message controls, reversed order reduces scores by a median of 0.105, compared with 0.137 in conversations. In contrast, increasing from two to six turns has almost no effect in the one-message controls (−0.007), compared with −0.044 in conversations. This distinction supports the use of matched controls to separate prompt arrangement from the additional effect of interaction. We will report the per-factor comparisons in Appendix R.

To address model scale, we evaluated Qwen2.5-72B and three frontier API models. Significant design sensitivity persists for Qwen2.5-72B on math and tool selection, Claude Haiku 5.5 on both tasks, Claude Sonnet 5.5 on tool selection and database, and GPT-5.5 on tool selection (all p < 0.001). The exception is Sonnet on math, where single-message accuracy is already 0.96 and the design effect is not distinguishable from noise (p = 0.57). Thus, sensitivity is not limited to smaller open-weight models, although ceiling performance may constrain its magnitude. We will update Sections 4.1 and 7 accordingly.

Finally, our contribution differs from Laban et al. in its focus on evaluation reliability rather than demonstrating performance degradation. We systematically measure how evaluator-selected conversation designs affect scores, model comparisons, and mitigation gains while holding the task content and checker fixed. Our matched one-message controls further distinguish prompt-arrangement effects from interaction effects. In particular, we find a median held-out design contrast of 0.12, design-dependent mitigation gains, and that Huang et al.'s user-turn-only method closely matches the corresponding one-message control. We will clarify these distinctions from prior work in the manuscript.

Thank you for raising these concerns. The additional experiments have helped us strengthen the evidence and clarify the scope of our conclusions.

We wanted to follow up on our response posted on November 12. As the discussion closes on November 18 (AoE), please let us know if any concerns remain or if further clarification would be helpful so we can respond before the deadline.

If we have adressed all your concerns, please update your rating, thank you for your time and feedback.

We have uploaded a revised PDF incorporating the changes described in our response, with the main revisions highlighted in blue. As the discussion closes on November 18 (AoE), please let us know if you have any further questions or if any points still need clarification or revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.