- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work addresses error localization of LLMs in multi-turn conversations. A metric LOCI and an adaptive repair policy were proposed. Experimental results on 20 Questions (20Q) game (in four main domains including Animals, Food, Objects, and Places) and CUB-200-2011 (a dataset of 200 bird species) show empirical evidence of the improved error localization and recovery.
Critical strengths
- The work targets an important issue that is still not fully tackled by recent approaches.
Critical weaknesses
- The experiments were based on Llama 3.1 8B. For the 2027 conference, more recent models such as GPT, Gemini, Qwen, and Gemma could be adopted in the experiments.
Questions for authors
- The experiment in this version focused on Llama 3.1 8B only. I wonder the results of the more recent, larger LLMs.
2 comments
Dear Reviewer fB6T,
We have uploaded a revised version of the paper, and some references in our response have changed: the Qwen3-8B results are now in Table A11 (previously A10) and mentioned at line ~408 (previously ~397), and the sentence on longer conversations is now at line ~406 (previously 395).
Your review raised one main concern: evaluation beyond Llama 3.1 8B. We hope our response addressed it with results for three additional models: Qwen3-8B, gpt-oss-20b, and Llama 3.1 70B. In all three cases, LOCI achieves the lowest recovery cost. As the discussion phase comes to an end, we would appreciate your feedback on whether this resolves your concern, or if there is anything else we can clarify.
Response to Reviewer fB6T
We thank the reviewer for recognizing the importance of the problem we study and that it has not yet been fully addressed by recent approaches.
W1/Q1: Evaluation of additional models
We agree that evaluation with other model families is important. The submitted paper already includes experiments with Qwen3-8B (Section 4.2.1, line ~397; Table A10). To further address the reviewer's request, we have now also evaluated gpt-oss-20b. We report both results below.
Qwen3-8B results (already in the paper). For ease of reading, we reproduce the results from Appendix B.2.3 below.
For Qwen, we see that LOCI achieves lower average recovery costs than all the baselines in every domain, consistent with the results for Llama.
Additional evaluation with gpt-oss-20b. We also ran the free-form recovery experiment with gpt-oss-20b (low reasoning effort). Each model generates its own failure cohort, so the relevant comparisons are between repair methods within each model. We report the results below.
For GPT, an interesting observation is that the model does not usually use the full budget of 20 questions before making a guess (mean initial questions: 7.15 for GPT, vs. 19.62 and 20 for Qwen and Llama, respectively). This leaves fewer prior updates to localize and possibly a smaller margin for LOCI improvement (see line 395 "This suggests that LOCI may be more effective in longer conversations"). Nonetheless, LOCI also achieves the lowest observed average recovery cost among the evaluated methods.
Stronger LLM as an error selector. Besides the conversation generation task, we have also added a stronger model as an error selector, Llama 3.1 70B. That is, we use this model to diagnose corrupted inferences in the conversation and use it as a baseline for comparison with LOCI. We found that it takes 18.66 recovery interactions on average to repair conversations generated by Llama 3.1 8B, while LOCI requires 12.47 interactions on the same conversations.