acceptodds
Under review as a conference paper at ICLR 2027

Isolated Consistency: Improve Multi-Turn Reasoning via Turn-Level Isolation

Abstract

Multi-turn conversation lets users specify a task gradually, but language models often fail even after all requirements have been provided. One difficulty is that the model starts answering before it has the full task. These early responses remain in the conversation and can shape later answers, even when new requirements invalidate their assumptions. Restating the complete task clarifies what the model should solve, but does not tell it which earlier responses to trust. These responses may offer useful intermediate reasoning while also carrying mistaken assumptions, so neither keeping nor discarding them all is an obvious choice. We observe that responses containing errors often make up only a minority of assistant turns. This suggests that isolating earlier responses can keep these errors out of most candidate contexts, rather than exposing every attempt to them. Instead of training a selector to decide what to retain, we introduce Isolated Consistency (IC), a training-free method compatible with both white-box and black-box models that revisits the task with each earlier response in isolation and then combines the resulting answers through voting. Concretely, IC generates one candidate answer for each earlier assistant turn. Each candidate uses all user messages and the supplied clean recap, but only the assistant response from that turn, with all other responses masked. IC then groups equivalent answers and selects the most common one. Across seven models and four analytical tasks, IC improves recovery beyond the strongest evaluated multi-turn baseline, raising macro-average accuracy by 3.19 percentage points with the same number of candidates and voting rule. These gains extend even to GPT-5.5, where IC improves accuracy by 25.87 points over standard multi-turn prompting and by 3.54 points over the strongest baseline, showing that stronger models still benefit from reconsidering which earlier responses to use.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Turn Conversation, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4kB6OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In multi-turn conversations, models often start answering before the user has given every requirement. The proposed method, Isolated Consistency (IC), keeps all of the user's turns but shows each new attempt only one earlier assistant response, then votes across the results. Across seven models and four analytical tasks, IC beat the best multi-turn baseline often, and the vote also helps users spot unreliable responses worth inspecting more.

Critical strengths

--the paper is quite a comprehensive study in the number of models (7 models from various model families) and previous methods (see table 1) it compares to.

--the paper is well-written and very easy for readers to follow; each section contains a clear message.

Critical weaknesses

--the paper uses only the synthetic sharded multi-turn conversations from lost-in-conversation (Laban et al., 2026), which are synthetically sharded multi-turn conversations. it would be interesting to see this applied to real in-the-wild conversations (e.g., wild chat).

--the method proposed is quite similar to the AO method in Huang et al. but requires k times the compute (where k is the number of assistant turns in the conversation). It would be great to see an argument for why including one round of the assistant-side responses, then taking the majority vote answer, makes more (logical) sense compared to omitting all assistant responses. aka, what is the value in keeping only one of the assistant responses (it must seem strange for the assistant to see only one of its previous responses while omitting all others), and then pooling across the turns?

Questions for authors

--what marker did you use for the turns you left out?

--how were the assistant-side responses sampled (e.g., were they sampled with all of the previous user and assistant responses in context, or only the user responses)?

--do you have a sense for how these results would hold in real-world multi-turn conversations?

6 comments

Thank you for recognizing our work and for your careful review. We provide detailed responses below.

W1/Q3: the paper uses only the synthetic sharded multi-turn conversations from lost-in-conversation (Laban et al., 2026), which are synthetically sharded multi-turn conversations. it would be interesting to see this applied to real in-the-wild conversations (e.g., wild chat). Do you have a sense for how these results would hold in real-world multi-turn conversations?

Response:
Thank you for this thoughtful suggestion. Evaluating IC beyond synthetically sharded conversations is an important next step, and we appreciate the opportunity to test it.

Our original experiments follow the controlled setting introduced by the ICLR 2026 Outstanding Paper, LLMs Get Lost in Multi-Turn Conversation [1]. Its synthetic sharding keeps the underlying task fixed, making it possible to isolate the effect of assistant history. Real conversations are less structured: users may answer the assistant’s questions, refer to its earlier results, or ask it to revise content that it generated.

We therefore added experiments with both GPT-4o-mini and GPT-5.5 on three context-dependent multi-turn benchmarks:

  • ShARC [2] evaluates conversational rule reasoning in which later user responses depend on clarification questions asked by the assistant.
  • CoSQL [3] evaluates conversational text-to-SQL in which follow-up requests may depend on earlier queries, results, and dialogue context.
  • MultiChallenge [4] evaluates realistic multi-turn interactions such as instruction retention, iterative editing, and self-coherence.
ModelMethodShARCCoSQLMultiChallengeAverage
GPT-4o-miniRestart58.0129.210.0029.08
Restart-vote54.1430.340.0028.16
AO-vote55.8042.700.0032.83
Assistant Masked Plus Recap46.4138.208.3330.98
IC69.6144.948.3340.96
GPT-5.5Restart67.4039.338.3338.35
Restart-vote62.9837.088.3336.13
AO-vote71.2729.218.3336.27
Assistant Masked Plus Recap75.6937.088.3340.37
IC80.1147.1916.6747.99

IC achieves the highest overall accuracy on both models. Averaged across GPT-4o-mini and GPT-5.5, IC reaches 44.48%, compared with 33.71% for Restart, 32.15% for Restart-vote, 34.55% for AO-vote, and 35.67% for Assistant Masked Plus Recap. These correspond to improvements of 10.76, 12.33, 9.92, and 8.80 percentage points, respectively. The consistent gains on GPT-5.5 further show that the benefit is not limited to a smaller model.

WildChat is valuable for studying naturally occurring conversations, but its conversations generally do not provide gold task outcomes for controlled accuracy comparisons. The three added benchmarks provide a useful bridge: they preserve reproducible evaluation while introducing the clarification, reference, and revision dependencies found in real interactions. The results suggest that IC remains useful beyond synthetic sharding, particularly when later requests depend on content created earlier in the conversation. We have added experiments on the context-dependent ShARC, CoSQL, and MultiChallenge benchmarks in Section 3.3, Table 2, and Appendix B.1 of the revised paper, highlighted in blue.

References

[1] Laban P, Hayashi H, Zhou Y, et al. Llms get lost in multi-turn conversation[C]//International Conference on Learning Representations. 2026, 2026: 54738-54778. [2] Saeidi et al. Interpretation of Natural Language Rules in Conversational Machine Reading. EMNLP, 2018.
[3] Yu et al. CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases. EMNLP-IJCNLP, 2019.
[4] Deshpande et al. MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs. Findings of ACL, 2025.

W2: the method proposed is quite similar to the AO method in Huang et al. but requires k times the compute (where k is the number of assistant turns in the conversation). It would be great to see an argument for why including one round of the assistant-side responses, then taking the majority vote answer, makes more (logical) sense compared to omitting all assistant responses. aka, what is the value in keeping only one of the assistant responses (it must seem strange for the assistant to see only one of its previous responses while omitting all others), and then pooling across the turns?

Response:
Thank you for raising this important question. AO and IC begin from the same concern that earlier assistant responses can mislead later answers, but they use the assistant history differently. AO retains the user turns and dialogue structure while replacing the content of every earlier assistant response with [Response provided]. IC instead retains one different assistant response in each candidate view.

Figure 2 motivates this choice. Only 10.35–21.75% of assistant turns are judged to contribute to the final failure. Replacing every response therefore removes not only the few harmful turns, but also any useful content introduced by the assistant. In many conversations, that content becomes necessary for understanding a later request:

Example 1: Referring to an assistant-proposed option

User: Any suggestions for a weekend getaway?
Original assistant response, replaced by AO with [Response provided]:

  1. Spend a weekend exploring Paris.
  2. Visit Amsterdam and cruise along its canals.
  3. Relax in a countryside cottage in the Cotswolds.

User: The first one sounds great. Could you put together a two-day itinerary?

Without the response content, “the first one” no longer identifies a destination.


Example 2: Modifying assistant-generated content

User: Could you write a Python function to read a CSV file?
Original assistant response, replaced by AO with [Response provided]: [Provides a 20-line Python implementation.]
User: I’m getting a UnicodeDecodeError on line 12. Could you fix it without changing the rest of the function?

Without the response content, neither “line 12” nor “the rest of the function” has a recoverable referent.

These examples illustrate the central trade-off. Keeping the full assistant history preserves useful information but also exposes every new attempt to every earlier mistake. AO avoids those mistakes by removing all assistant-side content, but it may also remove information needed by the current request.

IC takes a middle path without first predicting which responses are useful. It constructs one view per assistant turn, so every earlier response remains available in one view, while no response appears in every view. As a result, an error confined to one turn can influence only its associated candidate, whereas useful information can still contribute to the candidate pool. Voting then favors answers that remain stable across these different views.

To separate this mechanism from the additional computation, we added AO-vote. AO-vote generates KK candidates from the same AO context and uses exactly the same answer-grouping and plurality-voting rule as IC. The two methods therefore use the same number of model calls; they differ in whether the candidates all see placeholders or each retain a different assistant response.

As reported in our response to W1/Q3, IC achieves 40.96% average accuracy on GPT-4o-mini, compared with 32.83% for AO-vote, an improvement of 8.13 percentage points. On GPT-5.5, IC achieves 47.99%, compared with 36.27% for AO-vote, an improvement of 11.72 points. Averaged across the two models, IC outperforms AO-vote by 9.92 points under the same candidate budget and voting rule.

The original AO method requires one model call, whereas IC requires KK. The KK calls are independent and can run in parallel, and the compute-matched AO-vote comparison shows what the additional inference achieves: the gain comes from preserving different pieces of assistant-side information, rather than from repeated sampling alone. We have added a compute-matched AO-vote comparison and explained the value of retaining one assistant response per isolated view in Section 3.3 and Appendix B.1 of the revised paper, highlighted in blue.

Q1: what marker did you use for the turns you left out?

Response:
We use the literal text [MASKED] for omitted assistant responses. It is ordinary text rather than a model-specific mask token. Each IC view retains one original assistant response and places [MASKED] in every other original assistant-message slot, preserving all user messages, turn order, and dialogue structure. This procedure is already specified in Algorithm 1 and Appendix C.

Q2: how were the assistant-side responses sampled (e.g., were they sampled with all of the previous user and assistant responses in context, or only the user responses)?

Response:
The assistant-side responses used by IC are the original responses generated during the standard multi-turn conversation. At turn ii, the model receives the complete preceding history, including all earlier user and assistant messages:

$ A_i \sim p_\theta(\cdot \mid S,U_1,A_1,\ldots,U_i). $

They are therefore not generated from user-only context. After the original conversation is complete, IC constructs KK recovery views. View ii retains the original response AiA_i, replaces every other assistant response with [MASKED], and appends the same completed-task recap. The model then generates one new candidate answer from each view.

Thus, the historical assistant responses are generated with standard full-history conditioning; isolation is applied only when constructing the recovery views. We have clarified in Appendix B.1 of the revised paper that the original assistant responses are generated from the ordinary full conversation before turn-level isolation is applied, highlighted in blue.

Reviewer_qx9tOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study failures in conversations where a task's requirements are revealed incrementally. Earlier assistant responses may contain assumptions that become invalid as additional requirements arrive, yet these responses remain in the context and influence subseqent answers. The paper proposes the method Isolated Consistency which attempts to recover from this accumulated history after the complete task specification becomes available, by constructing multiple views of the completed conversation and does plurality voting over task specific answer groups. Evaluation is provided across four tasks from the Lost in Conversation framework.

Critical strengths
  1. The authors propose a simple intervention with minimal implementation requirement by retaining one historical assistant response and masking the others and voting.
  2. The improvements reported in Table 1 over SC has improvements in 24 out of 28 model task comparisons, with reasonable pp improvements which makes their results empirically strong and persuasive.
  3. The complete trajectories in Appendix B are interesting for follow-up work.
Critical weaknesses
  1. The paper is missing matched-K restart and all-assistant-masked-plus-recap baseline, without them the results do not establish why retaining isolated historical responses is preferable to removing assistant history entirely.
  2. Figure 2 motivates IC using rates of incorrect and failure contributing assistant turns, however I could not find a reproducible description of how these labels were assigned.
  3. The evals assume a clean and deterministically constructed task with a known recovery point, however these assumptions are not reasonable for realistic reconstruction errors.
Questions for authors

See weaknesses section above.

4 comments

Thank you for the careful review and valuable suggestions. We respond point by point below.

W1: The paper is missing matched-K restart and all-assistant-masked-plus-recap baseline, without them the results do not establish why retaining isolated historical responses is preferable to removing assistant history entirely.

Response:
Thank you for suggesting these controls. We added both requested baselines and distinguish them as follows:

  • Assistant Masked Plus Recap preserves all user turns and the original multi-turn structure, replaces every assistant response with [MASKED], appends the complete task recap, and generates one answer.
  • Restart-vote removes all earlier assistant responses, reconstructs the completed task from the user-side information in a fresh context, generates KK answers from that same context, and applies the same answer-grouping and plurality-voting rule as IC.
  • IC preserves the user turns, dialogue structure, and recap. It constructs KK views, each of which restores one different assistant response while keeping the others masked, and then applies plurality voting.

These controls examine two complementary questions. Assistant Masked Plus Recap tests whether the user turns, dialogue structure, and completed recap are sufficient after all assistant content is removed. Restart-vote controls the candidate budget and voting rule while solving the reconstructed task in a fresh context.

This distinction matters when a later request refers to content introduced by the assistant:

Example 1: Referring to an assistant-proposed option

User: Any suggestions for a weekend getaway?
Assistant, unavailable after Restart and replaced by [MASKED] in Assistant Masked Plus Recap:

  1. Spend a weekend exploring Paris.
  2. Visit Amsterdam and cruise along its canals.
  3. Relax in a countryside cottage in the Cotswolds.

User: The first one sounds great. Could you put together a two-day itinerary?

Without the assistant response, “the first one” no longer identifies a destination.


Example 2: Modifying assistant-generated content

User: Could you write a Python function to read a CSV file?
Assistant, unavailable after Restart and replaced by [MASKED] in Assistant Masked Plus Recap: [Provides a 20-line Python implementation.]
User: I’m getting a UnicodeDecodeError on line 12. Could you fix it without changing the rest of the function?

Without the assistant response, neither “line 12” nor “the rest of the function” has a recoverable referent.

In both examples, the assistant response contains information needed to interpret the later request. A recap built only from the user messages cannot recover that information. IC keeps each earlier response available in one candidate view without exposing every candidate to every potentially misleading response.

We further evaluated GPT-4o-mini and GPT-5.5 on three context-dependent multi-turn benchmarks:

  • ShARC [1] tests conversational rule reasoning, where user replies depend on clarification questions asked by the assistant.
  • CoSQL [2] tests conversational text-to-SQL, where follow-up queries may depend on earlier queries and system responses.
  • MultiChallenge [3] tests realistic interactions such as iterative editing and self-coherence, where later requests can directly depend on earlier assistant content.
ModelMethodShARCCoSQLMultiChallengeAverage
GPT-4o-miniRestart-vote54.1430.340.0028.16
AO-vote55.8042.700.0032.83
Assistant Masked Plus Recap46.4138.208.3330.98
IC69.6144.948.3340.96
GPT-5.5Restart-vote62.9837.088.3336.13
AO-vote71.2729.218.3336.27
Assistant Masked Plus Recap75.6937.088.3340.37
IC80.1147.1916.6747.99

Averaged across the two models, IC achieves 44.48%, compared with 35.67% for Assistant Masked Plus Recap, 32.15% for Restart-vote, and 34.55% for AO-vote. IC therefore improves over these controls by 8.80, 12.33, and 9.92 percentage points, respectively.

The comparison with Assistant Masked Plus Recap shows that preserving the user turns, dialogue structure, and recap does not fully replace the information contained in assistant responses. Meanwhile, the Restart-vote and AO-vote comparisons show that the improvement is not explained only by repeated sampling and voting. Together, these results support retaining assistant responses in isolated views rather than either exposing every candidate to the full assistant history or removing that history entirely. We have added Assistant-Masked+Recap and candidate-matched Restart-vote, together with their controlled comparison against IC, in Section 3.3, Table 2, and Appendix B.1 of the revised paper, highlighted in blue.

References

[1] Saeidi et al. Interpretation of Natural Language Rules in Conversational Machine Reading. EMNLP, 2018.
[2] Yu et al. CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databases. EMNLP-IJCNLP, 2019.
[3] Deshpande et al. MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs. Findings of ACL, 2025.

W2: Figure 2 motivates IC using rates of incorrect and failure contributing assistant turns, however I could not find a reproducible description of how these labels were assigned.

Response:

Thank you for pointing this out. Figure 2 is based on turn-level judgments generated by GPT-4o-mini, using a structured evaluation prompt with temperature set to 0. For each original multi-turn conversation, the judge receives the ordered user–assistant exchanges, the task-specific reference information, and the final benchmark outcome. Each assistant turn is annotated with two labels:

  1. Incorrect turn: A turn is labeled as incorrect if its response contains a material factual, logical, mathematical, coding, SQL, tool-calling, instruction-following, or context-retention error, or if it makes an unjustified premature answer. Importantly, correctness is assessed relative to the information available at that turn. The judge is explicitly instructed not to penalize earlier responses for failing to incorporate requirements revealed only in subsequent turns. Appropriate clarification questions and acknowledgments are therefore considered correct when information is insufficient.

  2. Failure-contributing turn: For conversations with an incorrect final outcome, an erroneous assistant turn is labeled as contributing to the failure if it introduced, repeated, or failed to correct a material error that plausibly influenced the final incorrect answer. Errors that were subsequently corrected need not receive this label. By definition, the contribution label is set to false for correctly judged turns and for conversations whose final outcome is correct.

The reported percentages are calculated from the resulting turn-level labels:

  • Incorrect-turn rate: Number of turns labeled incorrect divided by the total number of evaluated assistant turns.

  • Failure-contributing-turn rate: Number of turns labeled as contributing to the final error divided by the total number of evaluated assistant turns.

Figure 2 reports these statistics averaged across the four task categories.

For reproducibility, the complete implementation is included in our supplementary code:

  • prompts/sharded_turn_judge_system.txt: Exact annotation instructions and label definitions.

  • judge_sharded_turns.py: Judge inputs, GPT-4o-mini inference, output validation, and label normalization.

  • stats_sharded_turn_judge.py: Aggregation of turn-level labels into the reported statistics.

We have added the complete turn-level annotation criteria, denominator, and judge prompt in the Introduction, Appendix B.4, and Appendix E of the revised paper, highlighted in blue.

W3: The evals assume a clean and deterministically constructed task with a known recovery point, however these assumptions are not reasonable for realistic reconstruction errors.

Response:
Thank you for raising this concern. The original experiments follow the controlled setting introduced by the ICLR 2026 Outstanding Paper, LLMs Get Lost in Multi-Turn Conversation [4], where one complete task is divided into user-side shards. This makes the task and recovery point deterministic, allowing us to compare history-management methods while keeping the task itself fixed.

Many real conversations, however, cannot be reconstructed by simply joining the user messages. Later requests may depend on a clarification question, a database result, or content generated by the assistant. We therefore added experiments on three realistic, context-dependent multi-turn benchmarks:

  • ShARC contains clarification dialogues in which later user answers depend on questions asked by the assistant.
  • CoSQL contains conversational database queries in which later requests depend on earlier queries, results, and dialogue context.
  • MultiChallenge contains realistic multi-turn tasks such as iterative editing and self-coherence, where later instructions can directly refer to earlier assistant outputs.

These settings do not assume that the current task can be recovered by deterministically concatenating the user turns. As reported in our response to W1, IC achieves 40.96% average accuracy with GPT-4o-mini and 47.99% with GPT-5.5. Averaged across the two models, IC reaches 44.48%, compared with 35.67% for Assistant Masked Plus Recap and 32.15% for Restart-vote. The corresponding improvements are 8.80 and 12.33 percentage points.

The same pattern appears on the particularly challenging MultiChallenge benchmark. IC achieves 8.33% with GPT-4o-mini and 16.67% with GPT-5.5, compared with two-model averages of 8.33% for Assistant Masked Plus Recap and 4.17% for Restart-vote. These results suggest that the advantage of retaining isolated assistant responses extends across both a smaller model and a substantially stronger model.

The evaluation turn is specified by each benchmark, but IC does not require knowing which earlier turn caused the error or whether the current answer is correct. It only requires a point at which the system should produce the next answer, as in ordinary multi-turn inference. We have added evaluations on ShARC, CoSQL, and MultiChallenge to test context-dependent conversations beyond deterministic task reconstruction in Section 3.3 and Appendix B.1 of the revised paper, highlighted in blue.

References

[4] Laban P, Hayashi H, Zhou Y, et al. Llms get lost in multi-turn conversation[C]//International Conference on Learning Representations. 2026, 2026: 54738-54778.

Hi, just a gentle follow-up as the rebuttal ends in three days. We’d love to hear your thoughts. Thanks!

Reviewer_6pL9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the influence of mistakes in the previous turns in history of LLM. And how the models recovers from them in setting where user gradually reveal requirements. Earlier models turns may use unsupported assumptions that affect later outputs.

Isolated Consistency method creates masked versions from the same dialog history masking assistant turns. Then model generates from each version and the final answer is determined by voting.

Experiments cover several models and four tasks: math, code, sql, and function calling. IC beats self-consistency baseline with the same number of candidates.

Critical strengths
  • The main experiment isolates ICs effect in comparison ti self-consistency baseline: they both recieve the same complete task and generate the same number of candidates. This show that IC beats self-consistency on all reported models.
  • This simplicity of the method allows it to be used without additional training and on top of API-provided proprietary models.
Critical weaknesses
  • The benefit of keeping earlier assistant answers is unclear: starting a fresh conversation with the same complete task achieves higher average accuracy than IC (90 vs 87).
  • Test results are used to group candidate programs and select the final answer. The paper does not specify whether these tests are separate from those used to measure final accuracy. If the same tests serve both purposes, final evaluation information influences answer selection.
Questions for authors
  • How does IC compare with Restart when both generate the same number of candidate answers and use the same voting rule? What benefit remains from retaining earlier assistant responses?
  • Can you report paired confidence intervals for the differences between IC, self-consistency, and Restart?
6 comments

Thank you for your careful review. We respond point by point below.

W1/Q1: How does IC compare with Restart when both generate the same number of candidate answers and use the same voting rule? What benefit remains from retaining earlier assistant responses?

Response:
Thank you for raising this important question. We first clarify why retaining earlier assistant responses can be necessary, and then present a candidate-matched comparison between IC and Restart.

What benefit remains from retaining earlier assistant responses?
Restart is especially strong in Lost in Conversation because all user turns are pieces of one predefined task. Joining them together recovers the complete task, so the earlier assistant responses are not needed.

In more general conversations, however, assistant responses may contain part of the task state rather than merely recording earlier reasoning. Users frequently refer to options, questions, code, plans, or drafts produced by the assistant. Concatenating only the user messages can then leave the final request incomplete.

Example 1: Referring to an assistant-proposed option

User: Any suggestions for a weekend getaway?
Assistant — removed by Restart: Here are three options:

  1. Spend a weekend exploring Paris.
  2. Visit Amsterdam and cruise along its canals.
  3. Relax in a countryside cottage in the Cotswolds.

User: The first one sounds great. Could you put together a two-day itinerary?

After removing the assistant response, “the first one” no longer identifies a destination.


Example 2: Modifying assistant-generated content

User: Could you write a Python function to read a CSV file?
Assistant — removed by Restart: [Provides a 20-line Python implementation.]
User: I’m getting a UnicodeDecodeError on line 12. Could you fix it without changing the rest of the function?

After removing the assistant response, neither “line 12” nor “the rest of the function” has a recoverable referent.

Restart can handle these cases only by rewriting the request with information from the earlier assistant response. In other words, the assistant history is still needed; simply joining the user messages is not enough.

IC avoids making an all-or-nothing decision about the assistant history. Each candidate retains a different assistant response, so task-relevant content can remain available without exposing every candidate to every potentially misleading response. Agreement then favors answers that remain stable across these isolated views.

How does IC compare with candidate-matched Restart?
For a fair comparison, we added Restart-vote. Both methods generate KK answers and use the same voting rule. Restart-vote generates every answer from the same fresh prompt, whereas each IC candidate retains a different earlier assistant response. The only difference is the history shown to the candidates.

We evaluated both GPT-4o-mini and GPT-5.5 on three context-dependent multi-turn benchmarks:

  • ShARC [1] tests conversational rule reasoning. The assistant asks clarification questions, and later user replies such as “yes” or “no” cannot be understood without those questions.

  • CoSQL [2] tests conversational text-to-SQL. Users refine earlier queries and may refer to database results or explanations returned by the assistant.

  • MultiChallenge [3] tests realistic multi-turn interactions, including instruction retention, inference memory, iterative editing, and self-coherence. Several tasks directly depend on content generated earlier by the assistant.

ModelMethodShARCCoSQLMultiChallengeAverage
GPT-4o-miniRestart-vote54.1430.340.0028.16
AO-vote55.8042.700.0032.83
IC69.6144.948.3340.96
GPT-5.5Restart-vote62.9837.088.3336.13
AO-vote71.2729.218.3336.27
IC80.1147.1916.6747.99

With the same candidate budget and voting rule, IC outperforms Restart-vote by 12.33 percentage points on average across the two models. The gains are 12.80 points on GPT-4o-mini and 11.86 points on GPT-5.5.

We also compare IC with AO-vote, which uses KK candidates but replaces every earlier assistant response with a placeholder. IC outperforms AO-vote by 9.92 points on average, with gains of 8.13 points on GPT-4o-mini and 11.72 points on GPT-5.5. Because the candidate budget and voting rule are fixed, these results show that the improvement comes from retaining different individual assistant responses, rather than from generating more candidates. We have added the candidate-matched Restart comparison and the explanation of why retaining assistant responses matters in Section 3.3 and Appendix B.1 of the revised paper, highlighted in blue.

References

[1] Saeidi M, Bartolo M, Lewis P, et al. Interpretation of natural language rules in conversational machine reading[C]//Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. 2018: 2087-2097.

[2] Yu T, Zhang R, Er H, et al. Cosql: A conversational text-to-sql challenge towards cross-domain natural language interfaces to databases[C]//Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). 2019: 1962-1979.

[3] Deshpande K, Sirdeshmukh V, Mols J B, et al. Multichallenge: A realistic multi-turn conversation evaluation benchmark challenging to frontier llms[C]//Findings of the Association for Computational Linguistics: ACL 2025. 2025: 18632-18702.

W2: Test results are used to group candidate programs and select the final answer. The paper does not specify whether these tests are separate from those used to measure final accuracy. If the same tests serve both purposes, final evaluation information influences answer selection.

Response:
Thank you for pointing this out. We appreciate the opportunity to clarify the role of the unit tests and to provide an additional evaluation.

In the submitted experiments, we did not use reference programs or expose their source code to the model. Instead, candidate programs were executed against unit tests and grouped by their pass/fail patterns. The largest group was then selected. We viewed this as a verifier-assisted test-time procedure: the model receives no training or updates, and the verifier is used only to compare the behavior of generated candidates. This is similar in spirit to self-consistency, although textual answers are grouped by their content while programs are grouped by their execution behavior.

To make the separation between candidate selection and final evaluation especially clear, we reran all Code experiments without using any unit tests during voting. In this evaluation, candidates are grouped only by exact matches between their generated code strings. Unit tests are executed only after the final program has been selected, solely to measure its accuracy.

The verifier-free results are shown below:

ModelStandard multi-turn promptingICImprovement
GPT-4o-mini53.0062.00+9.00
Gemini-3.1-Flash-Lite73.0092.00+19.00
DeepSeek-V4-Flash78.9589.47+10.53
GPT-5.273.0083.00+10.00
Qwen3.5-Plus82.0090.00+8.00
Claude-Sonnet-4-681.8290.91+9.09
GPT-5.575.0080.00+5.00
Average73.8283.91+10.09

Without using unit-test outcomes during candidate selection, IC improves Code accuracy on every evaluated model. Its average accuracy increases from 73.82% to 83.91%, an improvement of 10.09 percentage points over standard multi-turn prompting.

These additional results show that the benefit of IC does not depend on access to an executable verifier during voting. We have clarified the verifier-assisted voting procedure and added the verifier-free exact-match Code results in Section 3.3 and Appendix B.2 of the revised paper, highlighted in blue.

Q2: Can you report paired confidence intervals for the differences between IC, self-consistency, and Restart?

Response:
Thank you for this suggestion. We computed paired 95% confidence intervals using aligned examples, so that each comparison evaluates the two methods on exactly the same instances. All differences are reported in percentage points.

ModelComparisonDifferencePaired 95% CI
GPT-4o-miniIC − SC+7.21[+4.26, +10.34]
IC − Restart−8.32[−12.56, −4.07]
SC − Restart−15.53[−19.96, −11.17]
Gemini-3.1-Flash-LiteIC − SC+4.84[+2.39, +7.47]
IC − Restart−1.88[−4.54, +0.76]
SC − Restart−6.72[−9.87, −3.59]

The paired intervals show that IC significantly improves over SC on both models. On GPT-4o-mini, IC improves over SC by 7.21 percentage points and reduces the gap to Restart from 15.53 to 8.32 points. On Gemini-3.1-Flash-Lite, IC improves over SC by 4.84 points and performs comparably to Restart, as the confidence interval for their difference includes zero.

These results support the benefit of varying the assistant history while also showing that Restart remains a strong reference when the complete task can be reconstructed perfectly. We have added the paired 95% confidence intervals and bootstrap procedure in Section 3.3 and Appendix B.3 of the revised paper, highlighted in blue.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your reviews and contributions. As we are now halfway through the discussion period, I would like to encourage you to carefully read the authors' responses and engage in further discussion where appropriate. Please also indicate whether the rebuttal has addressed your concerns or changed your assessment.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.