acceptodds
Under review as a conference paper at ICLR 2027

Where Did We Go Wrong? Localizing Errors to Repair Multi-Turn Conversations

Abstract

Large language model (LLM) assistants are increasingly used in multi-turn conversations, where they must progressively infer a user's intent from partial information revealed over time. This makes them vulnerable to early misinterpretations that can rule out the true intent and persist across turns, ultimately causing the interaction to fail. Recovering effectively from such failures requires identifying which earlier interpretations need revision while preserving the information that remains valid. We introduce Loci (Localization Of Corrupted Inferences), a framework for probabilistically localizing such errors and selectively repairing the conversational state. Given an observed failure, Loci reasons over possible user intents and assigns each prior update a posterior probability of having excluded the true intent. These probabilities rank likely error locations and quantify confidence in the localization. We then use this confidence in a repair policy that decides whether to retain, verify, or remove suspected updates, enabling selective repair of the conversational state. To evaluate Loci's ability to identify corrupted inferences, we first test it in a controlled setup with injected errors. We find that Loci improves localization over the baselines, with its advantage becoming larger when multiple errors occur. We then evaluate it in free-form conversations, where errors arise naturally, and measure the additional interactions needed to complete the task after an initial failure. Using localization confidence to guide repair, Loci reduces mean recovery cost by 46–79% relative to the strongest baseline across multiple domains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multiturn conversations, error localization, selective repair

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_fB6TOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses error localization of LLMs in multi-turn conversations. A metric LOCI and an adaptive repair policy were proposed. Experimental results on 20 Questions (20Q) game (in four main domains including Animals, Food, Objects, and Places) and CUB-200-2011 (a dataset of 200 bird species) show empirical evidence of the improved error localization and recovery.

Critical strengths
  • The work targets an important issue that is still not fully tackled by recent approaches.
Critical weaknesses
  • The experiments were based on Llama 3.1 8B. For the 2027 conference, more recent models such as GPT, Gemini, Qwen, and Gemma could be adopted in the experiments.
Questions for authors
  • The experiment in this version focused on Llama 3.1 8B only. I wonder the results of the more recent, larger LLMs.
2 comments
AuthorsOpenReview ↗

Response to Reviewer fB6T

We thank the reviewer for recognizing the importance of the problem we study and that it has not yet been fully addressed by recent approaches.

W1/Q1: Evaluation of additional models

We agree that evaluation with other model families is important. The submitted paper already includes experiments with Qwen3-8B (Section 4.2.1, line ~397; Table A10). To further address the reviewer's request, we have now also evaluated gpt-oss-20b. We report both results below.

Qwen3-8B results (already in the paper). For ease of reading, we reproduce the results from Appendix B.2.3 below.

MethodAverage ± SEAnimalsFoodObjectsPlaces
First + VERIFY21.22 ± 0.8421.39 ± 1.5720.45 ± 1.6625.30 ± 1.5017.71 ± 1.88
Last + VERIFY20.91 ± 1.1319.00 ± 1.4818.91 ± 1.3624.52 ± 1.6921.21 ± 3.56
Random + VERIFY22.17 ± 1.0321.22 ± 1.3619.95 ± 1.7525.94 ± 1.6421.57 ± 2.98
Counting + VERIFY20.18 ± 1.0320.61 ± 1.4519.23 ± 1.9324.52 ± 1.4716.36 ± 2.92
Restart47.57 ± 3.3643.11 ± 7.0343.18 ± 6.5570.79 ± 6.4633.21 ± 7.04
LOCI + VERIFY13.26 ± 0.8712.78 ± 1.489.86 ± 1.2120.12 ± 2.0810.29 ± 1.92
LOCI + MASK16.23 ± 1.7515.39 ± 3.8210.14 ± 2.3828.91 ± 4.5610.50 ± 3.14

For Qwen, we see that LOCI achieves lower average recovery costs than all the baselines in every domain, consistent with the results for Llama.

Additional evaluation with gpt-oss-20b. We also ran the free-form recovery experiment with gpt-oss-20b (low reasoning effort). Each model generates its own failure cohort, so the relevant comparisons are between repair methods within each model. We report the results below.

MethodAverageAnimalsFoodObjectsPlaces
First + VERIFY7.05 ± 1.099.00 ± 1.824.50 ± 0.3511.71 ± 3.543.00 ± 1.62
Last + VERIFY6.51 ± 0.867.50 ± 1.073.50 ± 0.3510.71 ± 2.784.33 ± 1.87
Random + VERIFY6.81 ± 1.098.12 ± 1.183.50 ± 0.3510.29 ± 3.895.33 ± 1.75
Counting + VERIFY4.64 ± 1.015.25 ± 1.462.50 ± 0.359.14 ± 3.781.67 ± 0.71
LOCI + VERIFY4.13 ± 0.614.12 ± 0.933.00 ± 0.707.71 ± 1.961.67 ± 0.71
LOCI + MASK3.98 ± 0.883.12 ± 0.873.00 ± 1.408.14 ± 2.961.67 ± 0.71
Restart19.00 ± 2.8013.88 ± 4.8722.00 ± 3.5136.14 ± 9.384.00 ± 0.80

For GPT, an interesting observation is that the model does not usually use the full budget of 20 questions before making a guess (mean initial questions: 7.15 for GPT, vs. 19.62 and 20 for Qwen and Llama, respectively). This leaves fewer prior updates to localize and possibly a smaller margin for LOCI improvement (see line 395 "This suggests that LOCI may be more effective in longer conversations"). Nonetheless, LOCI also achieves the lowest observed average recovery cost among the evaluated methods.

Stronger LLM as an error selector. Besides the conversation generation task, we have also added a stronger model as an error selector, Llama 3.1 70B. That is, we use this model to diagnose corrupted inferences in the conversation and use it as a baseline for comparison with LOCI. We found that it takes 18.66 recovery interactions on average to repair conversations generated by Llama 3.1 8B, while LOCI requires 12.47 interactions on the same conversations.

Dear Reviewer fB6T,

We have uploaded a revised version of the paper, and some references in our response have changed: the Qwen3-8B results are now in Table A11 (previously A10) and mentioned at line ~408 (previously ~397), and the sentence on longer conversations is now at line ~406 (previously 395).

Your review raised one main concern: evaluation beyond Llama 3.1 8B. We hope our response addressed it with results for three additional models: Qwen3-8B, gpt-oss-20b, and Llama 3.1 70B. In all three cases, LOCI achieves the lowest recovery cost. As the discussion phase comes to an end, we would appreciate your feedback on whether this resolves your concern, or if there is anything else we can clarify.

Reviewer_HwVgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors study recovery from multi turn conversaitonal failures caused by earlier interprtions that incorrectly exclude the user's intended target. They represent the assistant's working state as a sequence of constraints over a finite candidate set. They propose a framework LOCI which constructs a posterior over candidate targets using an update reliability model and rejection feedback. The paper derives a single error counting rule under uniformity assumptions and prove a corresponding rare error limit. Experiments include 20 questions tasks across four domains, CUB candidate set, free form generate conversations and a multiwoz derived recovery task.

Critical strengths
  1. The paper provides a useful distinction between conversation history and interpreted state. Section 2 separate the raw dialogue from the constraints the assistant has inferred from it, which makes the failure mechanism concrete. This is valuable to the community for studying recovery systematically.
  2. Their formulation is interesting, which accommodates multiple errors. Eq 1 provides a simple attribution rule where an update is suspicious when it excludes substantial posterior target mass.
  3. The experimental results are interesting and separate localization from intervention.
Critical weaknesses
  1. Eq 2 treats retaining a candidate as sufficient for the recorded update to be the unique correct interpration under that candidate, however it does not follow generally for the predicate substitutions used in Section 4.1. Replacement predicates are sampled from a target dependent eligible set making their likelihood candidate dependent.
  2. In section 4.2 the authors say that free form conversations lack ground truth error labels, however the experiment knows the hidden target and records each update's retained candidate set. Based on the section 2 definition whether an update excludes the target is observable to the evaluator, however in app B.2 they instead label disagreement between a judge's answer and the simulated user's answer, which measures a different event. This is inconsistent.
  3. The main are missing a different or stronger selector refreshed after verification, and a simple competitive adaptive baseline, currently their selectors are simple positional heuristics or their own error estimator which is weak.
Questions for authors

See weaknesses section above

3 comments
AuthorsOpenReview ↗

Response to Reviewer HwVg (Part 1)

We thank the reviewer for appreciating the distinction between conversation history and interpreted state, and for highlighting the experimental separation between localization and intervention. We address the concerns below.

W1: Likelihood model and controlled experiments

Choice of likelihood model. In free-form conversations, we do not have access to the distribution of the LLM's interpretation errors. For this reason, LOCI uses a simpler likelihood based on whether a candidate is compatible with each recorded update. We made the same choice in the controlled experiments, even though there the distribution of injected errors is known, so that LOCI does not rely on probabilities specific to the simulator. This being said, we agree with the reviewer that Eq. (2) does not necessarily give the exact likelihood of our controlled predicate-replacement process. In particular, a recorded predicate may retain a candidate without being the intended predicate for that candidate. We will clarify this modeling choice in Section 3.1 and its implementation in the controlled experiments in Section 4.1.

Experiment with candidate-dependent probabilities. In our controlled experiments we used the simpler choice qi(x)=1q_i(x)=1 for incompatible candidates. However, given that Eq. (2) allows the error likelihood to depend on the candidate target, one can also use candidate-dependent probabilities (assuming that these are known, which is not the case in free-form conversations). To address the reviewer's concern, we now also tested candidate-dependent probabilities based on the known replacement mechanism. Specifically, let Ei(x)E_i(x) be the set of eligible replacement predicates if xx were the true target; we then use qi(x)=1/∣Ei(x)∣q_i(x)=1/|E_i(x)| when the recorded replacement belongs to this set, and zero otherwise. We recomputed LOCI and a correspondingly weighted Counting baseline (which weights each candidate by how likely the observed predicate replacement would be if that candidate were the true target) on the same failed conversations, so that both methods use the same additional information.

We report the results under the Natural question policy, used for the main controlled experiments in Section 4.1, below:

Method (Hit@1 (%) / AP)p = 0.1p = 0.2p = 0.3p = 0.4p = 0.5
Weighted Counting63.5 ± 2.6 / 0.765 ± 0.01665.7 ± 2.2 / 0.758 ± 0.01468.6 ± 2.1 / 0.772 ± 0.01375.4 ± 1.8 / 0.811 ± 0.01177.0 ± 1.7 / 0.826 ± 0.010
Weighted LOCI69.0 ± 2.6 / 0.811 ± 0.01573.2 ± 2.1 / 0.822 ± 0.01377.3 ± 1.9 / 0.841 ± 0.01183.6 ± 1.6 / 0.871 ± 0.01085.2 ± 1.4 / 0.891 ± 0.008

Discussion. We see that LOCI still outperforms the correspondingly weighted Counting baseline at every error rate, on both metrics. Moreover, both methods benefit from using the candidate-dependent probabilities (e.g., at p=0.5p=0.5, LOCI's Hit@1 increases from 74.4% to 85.2%). Importantly, this shows that LOCI's advantage over Counting in our main controlled experiments does not depend on assuming constant error probabilities across candidates.

We will clarify the likelihood assumptions in Section 3.1 and include the additional results in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer HwVg (Part 2)

W2: Free-form error labels

Direct evaluation of localization. The reviewer is right that, as the hidden target and each stored candidate set are available, we can also evaluate localization by looking at the turns at which the true target gets eliminated by the (applied) constraints. Our original discussion conflated this target-exclusion criterion with the different question of whether a model-generated interpretation is semantically faithful to the user's response.

We have now also evaluated localization directly using the former criterion across all four domains and report the results below:

MethodHit@1 (%)AP
Random41.0 ± 5.80.506 ± 0.020
First14.4 ± 3.50.410 ± 0.014
Last52.0 ± 5.40.584 ± 0.023
Counting15.5 ± 3.60.411 ± 0.014
LOCI87.5 ± 3.90.802 ± 0.020

Results and consistency with recovery. Under this evaluation, LOCI again outperforms the baselines in every domain and on both metrics (with the gap wrt the baselines being greater than the one identified by the judge experiment in the appendix, which is however evaluated only on the subset of Animals in the 20q benchmark). Moreover, the results are consistent with the localization evaluation in the main body: in Table 1, column 20Q in the "Free-form LLM recovery" section, where we use the fixed-policy (with verify action) setup as a proxy for localization, we observe a similar gap in magnitude between LOCI and the baselines, and a similar ranking of the methods overall.

Rationale for the original evaluation. We originally used an independent judge because the free-form conversations do not have an externally supplied set of constraints and “ground-truth” answers, and the stored candidate sets are themselves generated by the assistant model (in other words, only the applied constraints are visible, but not the intended ones). Under this view, the judge labels can be seen as a test for whether the recorded answers are semantically (in)consistent with the hidden target, without treating the assistant's own classifications as the semantic ground truth. We agree that this measure is different from our formal definition of error, and this is why we preferred deferring the judge evaluation to the appendix while we instead used recovery cost under a fixed-policy setup in the main body.

We will clarify the distinction and add these results to the final version of the paper.

W3: Missing stronger/adaptive baselines

Baselines already included in the paper. We want to point out that, on top of the positional and random baselines, throughout the paper we also compare our method to Counting, which is simple, intuitive, and theoretically motivated (Section 3.2; Tables 1 and 2). We also evaluate asking the interacting LLM to identify its own errors (see the paragraph starting at line ~375, and Appendix B.2.2). We will make sure to make these results more visible in the main body.

This being said, we agree that the additional comparisons suggested by the reviewer are useful, and provide them below:

Stronger selectors with refreshed diagnoses. For this, we added a selector that is different and stronger, Llama 3.1 70B, with selection refreshed after verification, as requested. We also reran the previously mentioned Llama 3.1 8B experiment with refreshed selection. Both models output error probabilities for the remaining updates and select the highest-scoring update to verify.

We report the results below, and also report LOCI with the fixed VERIFY policy for ease of comparison, with all methods being evaluated on the same set of failed conversations as the other experiments in the paper.

Selector with fixed VERIFY policyRecovery interactions
Llama 3.1 8B (already in the paper)21.19 ± 1.20
Refreshed Llama 3.1 8B19.08 ± 0.35
Refreshed Llama 3.1 70B18.66 ± 0.39
LOCI12.47 ± 0.79

Allowing the diagnosing model to refresh after verification improves recovery by ~2 interactions on average. Increasing the model size further saves about half an interaction more. However, LOCI still outperforms the new baselines, requiring ~33% fewer interactions than the (refreshed) 70B selector.

An additional adaptive baseline. To provide an adaptive baseline, we ran the base model (Llama 3.1 8B) on its own conversations (similar to the self-diagnosing experiments above), but this time, we used the error probabilities to choose KEEP, VERIFY, or MASK. We provide the results below, and include adaptive LOCI for ease of comparison.

Adaptive methodRecovery interactions
Llama 3.1 8B13.50 ± 0.54
LOCI8.14 ± 0.96

Even though we see good improvement by making the selector adaptive, LOCI again outperforms the baseline, supporting the value of our score both under a fixed repair action and when thresholded to guide an adaptive policy.

Dear Reviewer HwVg,

Thank you again for your review. We wanted to let you know that we have uploaded a revised version with the additions from our response, including a clarification of the likelihood in Section 3.1, candidate-dependent results in Appendix B.5.2 (Table A17), target-exclusion localization in Appendix B.2.1 (Table A8), and the refreshed 8B/70B selectors + the adaptive LLM baseline in Appendix B.2.2 (Table A10). With this revision, some references in our original response have changed: the paragraph on the LLM as error selector now starts at line ~381 (previously ~375), and the judge-based evaluation is now in Table A9 (previously Table A8).

As the discussion phase comes to an end, please let us know whether our response addresses your concerns, or if there is anything else we can clarify.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1KTF (Part 1)

We thank the reviewer for appreciating our corruption score formulation and its theoretical connection to Counting, and for recognizing that the shared-VERIFY comparisons provide evidence for the localization method itself. We address the concerns below.

W1/Q1. Use of negative verification feedback

We agree that negative verification feedback can provide useful information. However, our implementation deliberately treats a negative verdict as masking, without using this additional information. We explain this choice below and provide an ablation that incorporates negative feedback.

Motivation for the original verification rule. Our choice reflects a broader view of verification than checking whether the target satisfies a constraint. For example, verification could involve consulting a stronger model or a tool, for which a negative response may mean that an interpretation is unsupported by the available evidence, rather than necessarily false (e.g., "it's the color of an apple" does not justify recording "it's red", even if the user’s intended object is indeed red, since an apple could also be green). In contrast, in our binary 20Q setup, a reliable negative verdict does rule out candidates satisfying the rejected constraint. We chose not to use this additional information, as it may not be available under more general forms of verification. Specifically, our original implementation retracts the rejected update without adding its complement as a new constraint. It also does not condition the posterior on the negative verdict elsewhere, so candidates ruled out by that verdict can retain posterior mass (while the original update remains available as fallible evidence for localization).

We stress that such choice was shared across all the evaluated methods (LOCI, and the baselines). That is, this is just an implementation choice about the same verification action that the other methods also use. For this reason, we believe the comparison is fair and does not invalidate our results.

Ablation using negative verification feedback. To make sure that there's no favorable interaction between LOCI and such choice, we report an additional experiment that implements the change suggested by the reviewer. That is, we re-ran the model allowing it to use information coming from negative verifications. We compare this to Counting, our theory-motivated baseline, under both implementations of the verify policy. All the rest of the setup remains the same as what is described in the main body. Results are reported below:

MethodOriginal feedback ruleNegative feedback used
Counting + VERIFY20.97±0.9118.18±0.37
LOCI + VERIFY11.97±0.6311.10±0.52
Adaptive LOCI8.14±0.966.44±0.77
LOCI + MASK11.10±1.4411.10±1.44

We see that under the new policy and conditional on the verify action being fixed, LOCI still outperforms Counting by a significant margin (18.18 -> 11.10 interactions). Moreover, allowing LOCI to use the adaptive policy further reduces recovery cost (11.10 -> 6.44 interactions). This ablation confirms the reviewer's intuition, and supports the robustness of our main findings.

We will make sure to clarify the original controller’s information restriction and include this ablation.

AuthorsOpenReview ↗

Response to Reviewer 1KTF (Part 2)

W2/Q2. Clarification on MultiWOZ results

Adaptive policy We agree that the MultiWOZ experiment provides limited evidence for the superiority of the adaptive policy over always-MASK. As already discussed in the paper, this appears to reflect the relatively low cost of recovering discarded information in this setting. We elaborate on this below.

i) Conversations in MultiWOZ are short (with an average of 2.54 applied constraints), making the verification action relatively expensive, since each check costs one interaction. At the same time, masking is relatively cheap here because the assistant can recover a discarded field by asking a targeted question (e.g., “Which area do you want?”), which can provide more information in one interaction compared to the binary answers in the 20Q setup. Indeed, Table A13 shows that adaptive repair saves 0.44 questions and wrong guesses compared with always-MASK, but requires 0.94 verification checks, resulting in 0.50 additional interactions overall.

ii) Moreover, for simplicity, our adaptive policy uses two distinct thresholds, assigning separate score regions to the available actions. Alternatively, one could define a broader policy class to also allow equal thresholds, so that the adaptive policy could adopt simpler action rules when those are preferable. This could allow the policy to avoid unnecessary actions in scenarios such as the observed one in MultiWOZ, where verification costs seem to exceed the interactions it saves.

Clarification of the experimental protocol. We agree that our wording may have been confusing. We initially conducted a preliminary analysis of the full dataset, including the test dialogues, which made us realize that dialogues with only one requirement-bearing turn made localization trivial. The only choice that was informed by this analysis was to only keep dialogues in which at least two turns introduce requirements (see lines ~407–409, or for more detail Appendix B.3.1), so that identifying an incorrect update requires choosing among multiple updates. Including the excluded one-update dialogues would make the methods look more similar (without changing their relative ranking), since all methods trivially achieve 100% localization accuracy when there is only one update to identify.

Importantly, we want to stress that, besides this filtering criterion, no other experimental choices were informed by the full dataset. That is, the repair policy was not modified, and thresholds were tuned exclusively on development dialogues and fixed for test evaluation. In that sense, the test set was not entirely untouched, but no test performance information was used to optimize the final method. We will clarify this distinction in Appendix B.3.

Dear Reviewers,

Thank you for your reviews and contributions. As we are now halfway through the discussion period, I would like to encourage you to carefully read the authors' responses and engage in further discussion where appropriate. Please also indicate whether the rebuttal has addressed your concerns or changed your assessment.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.