- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In multi-turn conversations, models often start answering before the user has given every requirement. The proposed method, Isolated Consistency (IC), keeps all of the user's turns but shows each new attempt only one earlier assistant response, then votes across the results. Across seven models and four analytical tasks, IC beat the best multi-turn baseline often, and the vote also helps users spot unreliable responses worth inspecting more.
Critical strengths
--the paper is quite a comprehensive study in the number of models (7 models from various model families) and previous methods (see table 1) it compares to.
--the paper is well-written and very easy for readers to follow; each section contains a clear message.
Critical weaknesses
--the paper uses only the synthetic sharded multi-turn conversations from lost-in-conversation (Laban et al., 2026), which are synthetically sharded multi-turn conversations. it would be interesting to see this applied to real in-the-wild conversations (e.g., wild chat).
--the method proposed is quite similar to the AO method in Huang et al. but requires k times the compute (where k is the number of assistant turns in the conversation). It would be great to see an argument for why including one round of the assistant-side responses, then taking the majority vote answer, makes more (logical) sense compared to omitting all assistant responses. aka, what is the value in keeping only one of the assistant responses (it must seem strange for the assistant to see only one of its previous responses while omitting all others), and then pooling across the turns?
Questions for authors
--what marker did you use for the turns you left out?
--how were the assistant-side responses sampled (e.g., were they sampled with all of the previous user and assistant responses in context, or only the user responses)?
--do you have a sense for how these results would hold in real-world multi-turn conversations?
6 comments
W2: the method proposed is quite similar to the AO method in Huang et al. but requires k times the compute (where k is the number of assistant turns in the conversation). It would be great to see an argument for why including one round of the assistant-side responses, then taking the majority vote answer, makes more (logical) sense compared to omitting all assistant responses. aka, what is the value in keeping only one of the assistant responses (it must seem strange for the assistant to see only one of its previous responses while omitting all others), and then pooling across the turns?
Response:
Thank you for raising this important question. AO and IC begin from the same concern that earlier assistant responses can mislead later answers, but they use the assistant history differently. AO retains the user turns and dialogue structure while replacing the content of every earlier assistant response with[Response provided]. IC instead retains one different assistant response in each candidate view.Figure 2 motivates this choice. Only 10.35–21.75% of assistant turns are judged to contribute to the final failure. Replacing every response therefore removes not only the few harmful turns, but also any useful content introduced by the assistant. In many conversations, that content becomes necessary for understanding a later request:
Example 1: Referring to an assistant-proposed option
User: Any suggestions for a weekend getaway?
Original assistant response, replaced by AO with[Response provided]:
- Spend a weekend exploring Paris.
- Visit Amsterdam and cruise along its canals.
- Relax in a countryside cottage in the Cotswolds.
User: The first one sounds great. Could you put together a two-day itinerary?
Without the response content, “the first one” no longer identifies a destination.
Example 2: Modifying assistant-generated content
User: Could you write a Python function to read a CSV file?
Original assistant response, replaced by AO with[Response provided]: [Provides a 20-line Python implementation.]
User: I’m getting aUnicodeDecodeErroron line 12. Could you fix it without changing the rest of the function?Without the response content, neither “line 12” nor “the rest of the function” has a recoverable referent.
These examples illustrate the central trade-off. Keeping the full assistant history preserves useful information but also exposes every new attempt to every earlier mistake. AO avoids those mistakes by removing all assistant-side content, but it may also remove information needed by the current request.
IC takes a middle path without first predicting which responses are useful. It constructs one view per assistant turn, so every earlier response remains available in one view, while no response appears in every view. As a result, an error confined to one turn can influence only its associated candidate, whereas useful information can still contribute to the candidate pool. Voting then favors answers that remain stable across these different views.
To separate this mechanism from the additional computation, we added AO-vote. AO-vote generates candidates from the same AO context and uses exactly the same answer-grouping and plurality-voting rule as IC. The two methods therefore use the same number of model calls; they differ in whether the candidates all see placeholders or each retain a different assistant response.
As reported in our response to W1/Q3, IC achieves 40.96% average accuracy on GPT-4o-mini, compared with 32.83% for AO-vote, an improvement of 8.13 percentage points. On GPT-5.5, IC achieves 47.99%, compared with 36.27% for AO-vote, an improvement of 11.72 points. Averaged across the two models, IC outperforms AO-vote by 9.92 points under the same candidate budget and voting rule.The original AO method requires one model call, whereas IC requires . The calls are independent and can run in parallel, and the compute-matched AO-vote comparison shows what the additional inference achieves: the gain comes from preserving different pieces of assistant-side information, rather than from repeated sampling alone. We have added a compute-matched AO-vote comparison and explained the value of retaining one assistant response per isolated view in Section 3.3 and Appendix B.1 of the revised paper, highlighted in blue.
Q1: what marker did you use for the turns you left out?
Response:
We use the literal text[MASKED]for omitted assistant responses. It is ordinary text rather than a model-specific mask token. Each IC view retains one original assistant response and places[MASKED]in every other original assistant-message slot, preserving all user messages, turn order, and dialogue structure. This procedure is already specified in Algorithm 1 and Appendix C.
Q2: how were the assistant-side responses sampled (e.g., were they sampled with all of the previous user and assistant responses in context, or only the user responses)?
Response:
The assistant-side responses used by IC are the original responses generated during the standard multi-turn conversation. At turn , the model receives the complete preceding history, including all earlier user and assistant messages:$ A_i \sim p_\theta(\cdot \mid S,U_1,A_1,\ldots,U_i). $
They are therefore not generated from user-only context. After the original conversation is complete, IC constructs recovery views. View retains the original response , replaces every other assistant response with
[MASKED], and appends the same completed-task recap. The model then generates one new candidate answer from each view.Thus, the historical assistant responses are generated with standard full-history conditioning; isolation is applied only when constructing the recovery views. We have clarified in Appendix B.1 of the revised paper that the original assistant responses are generated from the ordinary full conversation before turn-level isolation is applied, highlighted in blue.
Thank you for recognizing our work and for your careful review. We provide detailed responses below.
W1/Q3: the paper uses only the synthetic sharded multi-turn conversations from lost-in-conversation (Laban et al., 2026), which are synthetically sharded multi-turn conversations. it would be interesting to see this applied to real in-the-wild conversations (e.g., wild chat). Do you have a sense for how these results would hold in real-world multi-turn conversations?