- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies how retrieval-augmented generation (RAG) degrades when a question is delivered through a multi-turn conversation rather than all at once.
Building on the sharded-conversation protocol of Laban et al. (2026), the authors convert 750 multi-hop questions into dependency-ordered sequences of clues, revealed one per turn by an LLM user simulator. At every turn, the retriever is queried with either the latest user turn (CURRENT) or the concatenation of all user turns (HISTORY). The paper compares FULL (original question), CONCAT (all clues in one turn) and SHARDED (one clue per turn) across ten LLMs and seven retrievers plus a closed-book baseline.
Critical strengths
1. A matched single- versus multi-turn measurement for RAG. Each conversation is paired with its original question, so the cost of conversational delivery is measured directly rather than confounded with question difficulty. The study is broad (750 manually reviewed questions, ten LLMs, eight retrieval settings) and reports paired bootstrap intervals and an LLM-judge check (App. B.5).
2. Dependency-ordered sharding. The authors adapt the original order-insensitive shards to multi-hop chains, on the grounds that "multi-hop questions form reasoning chains that require sequential context" (Sec. 3.2), and make the construction rules explicit (P1–P6, App. A.1).
Critical weaknesses
1. The protocol and both conclusions are already established; the paper's addition is a retrieval step. The protocol is that of Laban et al.: "We keep its user simulation, response classification, and insert a retrieval step at every turn" (Sec. 3).
The paper's findings have been reported repeatedly in prior work:
- Degradation from multi-turn delivery, measured against matched single-turn versions. MT-Eval (Kwan et al., 2024) creates "single-turn versions of the 1170 multi-turn queries" and identifies "distance to relevant content" as a key factor. Laban et al. (2026) report the same pattern.
- LLMs failing to integrate information spread across turns. Multi-Turn-Instruct (Han et al., 2025) finds that models' "attention mechanisms fail to integrate multiple related instructions effectively".
- Underspecified users who reveal their needs incrementally. This setting is studied in UserBench (Qian et al., 2025) and Laban et al.
- The retrieval-side mechanisms. The paper's own introduction cites them: "Retrieval can be sensitive to how a query is phrased", and "irrelevant or distracting evidence can hurt downstream generation" (Sec. 1).
- Unreliability rather than aptitude. This is "Consistent with the original benchmark" (Sec. 5.4).
Confirming that known weaknesses co-occur in a new combination is not the same as identifying a new one. The paper does not explain what the combination of retrieval and multi-turn delivery produces beyond what these findings already predict. The one difference it claims, that consolidation fully recovers the gap for some retrievers, which "does not appear in the LLM-only setting of Laban et al." (Sec. 1), is explained only post hoc: "this preprocessing may make them less sensitive" (Sec. 5.3).
2. The two failure modes are named by component but are not isolated. The modes are defined by component: either the retrieval query is distorted, or "retrieval succeeds but the LLM fails" (Abstract). The decomposition, however, splits the gap by delivery condition. The paper itself states: "The formulation contrast (∆form) changes both the retrieval query and the assistant's input. The multi-turn contrast (∆turn) additionally distributes the same shards across turns and introduces repeated retrieval and answer opportunities" (Sec. 5.3).
- Lost in translation. It is measured without query rewriting. The paper acknowledges that "Conversational retrieval methods commonly rewrite context-dependent turns into standalone queries", but adopts "two transparent retrieval policies" instead (Sec. 3). The retrievers that are robust are precisely those that "first use an LLM to extract entities or keywords from the query before retrieval" (Sec. 5.3). The effect may therefore reflect the absence of rewriting rather than conversation.
- Lost in conversation. ∆turn mixes a gain from repeated retrieval with a loss from distributed delivery. It is negative, i.e. SHARDED scores higher than CONCAT, for four of the seven retrievers (Table 3). The complete-coverage analysis covers 24.4% of pairs (App. B.4), and earlier retrieved passages are not passed to later turns (, Sec. 3). How the LLM actually fails is not analyzed.
Questions for authors
Please see the weaknesses
4 comments
W2: Separating the two failure modes
The reviewer raises an important point about the distinction between locating where a gap appears and isolating its underlying cause. Our original FULL$\rightarrow$CONCAT$\rightarrow$SHARDED decomposition was intended primarily for the former. It shows whether degradation first appears when the original question is reformulated or when the same clues are subsequently distributed across turns. We agree, however, neither comparison isolates a single factor. We address their concerns below with additional experiments.
Isolating the Lost in Translation effect
Separating wording from retrieved evidence. We first isolate the roles of question wording and retrieved evidence by crossing the FULL and CONCAT questions with the evidence retrieved under the other condition. This gives the LLM either the FULL question with CONCAT evidence or the CONCAT question with FULL evidence.
| Retriever | FULL | CONCAT | FULL question + CONCAT evidence (FQCE) | CONCAT question + FULL evidence (CQFE) |
|---|---|---|---|---|
| BM25 | 39.8 | 30.1 | 30.1 | 39.8 |
| Dense | 48.1 | 42.3 | 43.6 | 46.4 |
| RAPTOR | 49.0 | 42.9 | 44.1 | 46.9 |
| HippoRAG | 49.1 | 47.9 | 50.3 | 47.0 |
| HippoRAG2 | 53.2 | 47.0 | 49.0 | 50.3 |
| LightRAG | 39.9 | 39.0 | 39.9 | 39.5 |
| ToG-2 | 49.2 | 46.9 | 48.3 | 47.0 |
| All retrievers | 46.9 | 42.3 | 43.6 | 45.3 |
The table reports BEST F1 $\times$100, averaging Gemma-4-31B, Qwen3-14B, and Luna 5.6 over all five datasets.
We observe that most of the average FULL$\rightarrow$CONCAT loss comes from differences in the retrieved evidence. Changing the retrieved evidence produces a 3.0–3.3 F1 effect on average (CQFE−CONCAT, FULL−FQCE), while changing only the question wording produces a smaller 1.3–1.6 point effect (FQCE−CONCAT, FULL−CQFE). The same three retrievers that were robust to formulation in Sec. 5.3 (HippoRAG, LightRAG, and ToG-2) are also robust in this experiment.
Testing query rewriting. The reviewer asks whether the formulation loss could simply reflect the absence of query rewriting. We test this out using the mtRAG rewriting prompt over the visible conversation history with GPT-OSS 20B, the strongest rewriter identified by MTRAG-UN [7]. We evaluate Qwen3-32B across all five datasets and six retrievers, changing only the rewritten query passed to the retriever.
| Retriever | FULL | HISTORY | CURRENT | Rewrite |
|---|---|---|---|---|
| BM25 | 33.0 | 25.9 | 28.1 | 28.3 |
| Dense | 43.5 | 35.2 | 30.0 | 31.5 |
| RAPTOR | 44.8 | 37.1 | 30.7 | 32.3 |
| HippoRAG | 42.9 | 38.5 | 34.1 | 34.3 |
| HippoRAG2 | 50.3 | 40.7 | 36.4 | 36.3 |
| LightRAG | 33.2 | 32.9 | 28.3 | 29.6 |
| All retrievers | 41.3 | 35.1 | 31.3 | 32.1 |
We find that rewriting helps, but only modestly. It improves the incomplete CURRENT query, while direct HISTORY remains substantially stronger. This shows a direct contrast with mtRAG [6]. In its episodic, turn-level setting, rewriting is the strongest retrieval strategy, and CURRENT performs better than HISTORY among the raw conversational inputs. In our setting, no individual turn contains the complete information need. Consequently, the rewriter must go a step further and resolve ambiguities across several incomplete clues. Resolving even one entity incorrectly at any point in the conversation can contaminate the retrieval query, whereas HISTORY preserves the original clues.
W2 (continued)
Isolating the Lost in Conversation effect
Stricter paired complete-coverage analysis. The reviewer notes that the paired complete-coverage analysis in App. B.4 pools passages across turns, even though earlier passages are not carried into later contexts. We therefore repeat the analysis using a stricter criterion that retains only cases where the final SHARDED turn contains every supporting passage.
| SHARDED Recall@5 | Pairs | Questions | FULL | SHARDED |
|---|---|---|---|---|
| 100% | 52,601 | 446 | 73.8 | 62.8 |
The degradation remains essentially unchanged even when the retrieval has already supplied all required supporting evidence at the final turn.
Retaining previously retrieved passages. The reviewer also points to whether the degradation could result from not retaining passages retrieved in earlier turns. Our original protocol retrieves a fresh context at every turn, following the retrieve-then-read setting used in conversational RAG benchmarks [6–9]. Nevertheless, retaining earlier passages is a plausible alternative explanation, so we tested it directly by appending all previously retrieved passages to the current context.
| LLM | FULL | HISTORY SHARDED | HISTORY Retain |
|---|---|---|---|
| Luna 5.6 | 63.7 | 57.5 | 57.9 |
| Qwen3-14B | 49.5 | 40.4 | 39.5 |
| Qwen3-32B | 51.7 | 46.0 | 46.4 |
| All three LLMs | 55.0 | 48.0 | 47.9 |
Retaining all previously retrieved passages leaves HISTORY performance essentially unchanged. Earlier-passage eviction therefore does not explain the degradation.
Quantifying failure causes through manual review. We also investigate how LLM fails by manually reviewing 200 failed SHARDED conversations in which every supporting passage had been retrieved. We find that, in 18% of the conversations, the required passages arrived in different turns and were never available together, which is the case most directly addressed by retaining earlier evidence. Beyond this, the failures were heterogeneous. In 32%, the LLM lost track of the original intent, typically answering only the latest clue or stopping one hop short. It misread a retrieved passage in 16%, interpreted the underspecified request differently in 12%, and had sufficient evidence but never committed to the answer in 7%.
We observe more than one of these failures in 42% of the conversations. This helps explain why the progressively specified setting remains difficult even after the required evidence has been retrieved. To answer correctly, an LLM has to preserve the user’s original goal as new constraints arrive, resolve ambiguities in both the request and retrieved evidence, and connect information revealed at different turns.
We thank the reviewer for prompting us to examine the causes of degradation more carefully.
References
[1] Kwan et al. MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models.
[2] Han et al. Can Language Models Follow Multiple Turns of Entangled Instructions?
[3] Qian et al. UserBench: An Interactive Gym Environment for User-Centric Agents.
[4] Laban et al. LLMs Get Lost in Multi-Turn Conversation.
[5] Chen et al. KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues.
[6] Katsis et al. mtRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems.
[7] Rosenthal et al. MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations.
[8] Ali et al. RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark.
[9] Cheng et al. CORAL: Benchmarking Multi-Turn Conversational Retrieval-Augmented Generation.
We are writing to kindly ask whether our rebuttal has addressed your concerns. Please let us know if any concerns remain.
We thank the reviewer for their careful reading and thoughtful comments.
W1: What retrieval adds beyond prior multi-turn findings
We agree that several phenomena discussed in our paper have precedents. Laban et al. [4] motivates our study of degradation when the same task is distributed across turns, while sensitivity to query phrasing and distracting evidence are established observations that we cite as background. Our contribution is to study what happens when retrieval operates throughout the progressive specification of one underspecified, multi-hop information need.
How the cited multi-turn work differs
MT-Eval [1] mainly measures how well a model continues to follow earlier rules or labels, or continues working from a document supplied in the conversation, as more turns intervene. Its single-turn counterparts place the relevant instructions and request together. It finds that models lose track of earlier content as the distance to it grows and that errors can propagate through later turns.
MultiTurnInstruct [2] gives the model several entangled instructions across turns, including privacy, triggering, and prioritization constraints, and measures how well it retains and reconciles them.
UserBench [3] studies whether a tool-using travel agent can uncover preferences revealed gradually through interaction and use structured search tools to select suitable options. It finds that agents can use the tools reliably while still uncovering only a small fraction of the user's preferences.
These papers study how an LLM or agent handles information, instructions, or preferences as conversational context accumulates. Their central challenges are memory, instruction reconciliation, and preference elicitation. They do not study how the progressive interpretation of an incomplete information need changes the external evidence retrieved from a corpus.
Relationship to conversational RAG benchmarks
KnowMT-Bench, mtRAG, MTRAG-UN, RECOR, and CORAL [5–9] are more closely related because they evaluate retrieval during conversation. Their tasks, however, are largely episodic: a turn typically introduces an information need with its own answer or relevance judgment, even when that turn depends on earlier context.
Our turns instead mimic a user progressively adding constraints to one final information need. No individual turn states the complete question, so retrieval must operate while the target is still ambiguous and incomplete. This can lead retrieval toward the wrong entity or supporting passage early in the conversation. This requires handling ambiguity, repeated retrieval, and evidence integration as the information need becomes clearer.
As far as we know, ours is the first benchmark to systematically evaluate this setting across a diverse range of widely established RAG designs, pairing an ideal standalone multi-hop question with a progressively specified conversational version.
Distance to relevant content. The reviewer also points to MT-Eval's finding that performance declines as relevant material becomes more distant in the conversation. We therefore grouped our questions by number of user turns and measured the paired FULL–SHARDED gap across all five datasets.
BEST F1 is reported $\times$100.
We observe no increase in degradation as conversations become longer, unlike the distance-dependent growth reported by MT-Eval.
Additional findings from our benchmark. We also observe retrieval-specific behavior that the cited LLM-only studies cannot measure. Findings such as strong retrievers and LLMs still struggling in conversation, and different retrievers responding differently to underspecified queries and repeated retrieval, are unique to our setting.
Query preprocessing and conversational robustness. The reviewer also asks whether our explanation based on query preprocessing is post hoc. In Sec. 5.3, we found that retrievers which first use an LLM to extract entities or keywords from the query (HippoRAG, LightRAG, and ToG-2) tend to be more robust to conversational reformulation than retrievers that operate directly on the query text or embedding. We arrived at this explanation after seeing the same retriever grouping across multiple analyses: FULL$\rightarrow$CONCAT answer degradation (Sec. 5.3), retrieved-evidence recall (Sec. 5.4), recovery under RECAP and SNOWBALL (App. B.1), and the evidence-swap experiment in W2. Across these experiments, the same three retrievers consistently stand out as more robust to the conversational setting. This is why we believe query preprocessing is an important explanation for this robustness, although we concede that our experiments do not isolate it as the sole cause.