acceptodds
Under review as a conference paper at ICLR 2027

Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAG

Abstract

When conversing with large language models (LLMs), users often begin with a simple question and build towards a multi-hop question through follow-up turns. Retrieval-augmented generation (RAG) and its graph-based variant (GraphRAG) have become the dominant approaches for grounding LLM responses in external evidence, yet both are evaluated almost exclusively on single-turn, fully specified queries. We systematically investigate this evaluation mismatch through a large-scale simulation study. Building on prior work on multi-turn LLM evaluation, we transform questions from multi-hop question answering (QA) benchmarks into underspecified conversations and evaluate ten LLM assistants with eight retrieval systems across 1.5 million simulated conversations. Our findings reveal that multi-turn interaction causes widespread performance degradation, incurring relative performance drops of up to 21% and increasing unreliability by 47%, making RAG systems simultaneously less accurate and less reliable. We identify two distinct failure modes behind this degradation. Systems are either lost in translation, where conversational rephrasing distorts the retrieval query, or lost in conversation, where retrieval succeeds but the LLM fails to synthesize evidence distributed across turns.

Primary area:
datasets and benchmarks
Keywords:
multi-turn, underspecification, LLM simulation, retrieval-augmented generation, GraphRAG, multi-hop question answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_TVh1OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies how retrieval-augmented generation (RAG) degrades when a question is delivered through a multi-turn conversation rather than all at once.

Building on the sharded-conversation protocol of Laban et al. (2026), the authors convert 750 multi-hop questions into dependency-ordered sequences of clues, revealed one per turn by an LLM user simulator. At every turn, the retriever is queried with either the latest user turn (CURRENT) or the concatenation of all user turns (HISTORY). The paper compares FULL (original question), CONCAT (all clues in one turn) and SHARDED (one clue per turn) across ten LLMs and seven retrievers plus a closed-book baseline.

Critical strengths

1. A matched single- versus multi-turn measurement for RAG. Each conversation is paired with its original question, so the cost of conversational delivery is measured directly rather than confounded with question difficulty. The study is broad (750 manually reviewed questions, ten LLMs, eight retrieval settings) and reports paired bootstrap intervals and an LLM-judge check (App. B.5).

2. Dependency-ordered sharding. The authors adapt the original order-insensitive shards to multi-hop chains, on the grounds that "multi-hop questions form reasoning chains that require sequential context" (Sec. 3.2), and make the construction rules explicit (P1–P6, App. A.1).

Critical weaknesses

1. The protocol and both conclusions are already established; the paper's addition is a retrieval step. The protocol is that of Laban et al.: "We keep its user simulation, response classification, and insert a retrieval step at every turn" (Sec. 3).

The paper's findings have been reported repeatedly in prior work:

  • Degradation from multi-turn delivery, measured against matched single-turn versions. MT-Eval (Kwan et al., 2024) creates "single-turn versions of the 1170 multi-turn queries" and identifies "distance to relevant content" as a key factor. Laban et al. (2026) report the same pattern.
  • LLMs failing to integrate information spread across turns. Multi-Turn-Instruct (Han et al., 2025) finds that models' "attention mechanisms fail to integrate multiple related instructions effectively".
  • Underspecified users who reveal their needs incrementally. This setting is studied in UserBench (Qian et al., 2025) and Laban et al.
  • The retrieval-side mechanisms. The paper's own introduction cites them: "Retrieval can be sensitive to how a query is phrased", and "irrelevant or distracting evidence can hurt downstream generation" (Sec. 1).
  • Unreliability rather than aptitude. This is "Consistent with the original benchmark" (Sec. 5.4).

Confirming that known weaknesses co-occur in a new combination is not the same as identifying a new one. The paper does not explain what the combination of retrieval and multi-turn delivery produces beyond what these findings already predict. The one difference it claims, that consolidation fully recovers the gap for some retrievers, which "does not appear in the LLM-only setting of Laban et al." (Sec. 1), is explained only post hoc: "this preprocessing may make them less sensitive" (Sec. 5.3).

2. The two failure modes are named by component but are not isolated. The modes are defined by component: either the retrieval query is distorted, or "retrieval succeeds but the LLM fails" (Abstract). The decomposition, however, splits the gap by delivery condition. The paper itself states: "The formulation contrast (∆form) changes both the retrieval query and the assistant's input. The multi-turn contrast (∆turn) additionally distributes the same shards across turns and introduces repeated retrieval and answer opportunities" (Sec. 5.3).

  • Lost in translation. It is measured without query rewriting. The paper acknowledges that "Conversational retrieval methods commonly rewrite context-dependent turns into standalone queries", but adopts "two transparent retrieval policies" instead (Sec. 3). The retrievers that are robust are precisely those that "first use an LLM to extract entities or keywords from the query before retrieval" (Sec. 5.3). The effect may therefore reflect the absence of rewriting rather than conversation.
  • Lost in conversation. ∆turn mixes a gain from repeated retrieval with a loss from distributed delivery. It is negative, i.e. SHARDED scores higher than CONCAT, for four of the seven retrievers (Table 3). The complete-coverage analysis covers 24.4% of pairs (App. B.4), and earlier retrieved passages are not passed to later turns (Ht=(u1,a1,…,at−1,ut)H_t = (u_1, a_1, \ldots, a_{t-1}, u_t), Sec. 3). How the LLM actually fails is not analyzed.
Questions for authors

Please see the weaknesses

4 comments

We thank the reviewer for their careful reading and thoughtful comments.

W1: What retrieval adds beyond prior multi-turn findings

We agree that several phenomena discussed in our paper have precedents. Laban et al. [4] motivates our study of degradation when the same task is distributed across turns, while sensitivity to query phrasing and distracting evidence are established observations that we cite as background. Our contribution is to study what happens when retrieval operates throughout the progressive specification of one underspecified, multi-hop information need.

How the cited multi-turn work differs

MT-Eval [1] mainly measures how well a model continues to follow earlier rules or labels, or continues working from a document supplied in the conversation, as more turns intervene. Its single-turn counterparts place the relevant instructions and request together. It finds that models lose track of earlier content as the distance to it grows and that errors can propagate through later turns.

MultiTurnInstruct [2] gives the model several entangled instructions across turns, including privacy, triggering, and prioritization constraints, and measures how well it retains and reconciles them.

UserBench [3] studies whether a tool-using travel agent can uncover preferences revealed gradually through interaction and use structured search tools to select suitable options. It finds that agents can use the tools reliably while still uncovering only a small fraction of the user's preferences.

These papers study how an LLM or agent handles information, instructions, or preferences as conversational context accumulates. Their central challenges are memory, instruction reconciliation, and preference elicitation. They do not study how the progressive interpretation of an incomplete information need changes the external evidence retrieved from a corpus.

Relationship to conversational RAG benchmarks

KnowMT-Bench, mtRAG, MTRAG-UN, RECOR, and CORAL [5–9] are more closely related because they evaluate retrieval during conversation. Their tasks, however, are largely episodic: a turn typically introduces an information need with its own answer or relevance judgment, even when that turn depends on earlier context.

Our turns instead mimic a user progressively adding constraints to one final information need. No individual turn states the complete question, so retrieval must operate while the target is still ambiguous and incomplete. This can lead retrieval toward the wrong entity or supporting passage early in the conversation. This requires handling ambiguity, repeated retrieval, and evidence integration as the information need becomes clearer.

As far as we know, ours is the first benchmark to systematically evaluate this setting across a diverse range of widely established RAG designs, pairing an ideal standalone multi-hop question with a progressively specified conversational version.

Distance to relevant content. The reviewer also points to MT-Eval's finding that performance declines as relevant material becomes more distant in the conversation. We therefore grouped our questions by number of user turns and measured the paired FULL–SHARDED gap across all five datasets.

Planned user turnsQuestionsFULLSHARDEDDegradation
224041.537.0+4.5
336940.935.4+5.5
412824.421.6+2.8
51333.634.0−0.4
All75038.133.5+4.6

BEST F1 is reported $\times$100.

We observe no increase in degradation as conversations become longer, unlike the distance-dependent growth reported by MT-Eval.

Additional findings from our benchmark. We also observe retrieval-specific behavior that the cited LLM-only studies cannot measure. Findings such as strong retrievers and LLMs still struggling in conversation, and different retrievers responding differently to underspecified queries and repeated retrieval, are unique to our setting.

Query preprocessing and conversational robustness. The reviewer also asks whether our explanation based on query preprocessing is post hoc. In Sec. 5.3, we found that retrievers which first use an LLM to extract entities or keywords from the query (HippoRAG, LightRAG, and ToG-2) tend to be more robust to conversational reformulation than retrievers that operate directly on the query text or embedding. We arrived at this explanation after seeing the same retriever grouping across multiple analyses: FULL$\rightarrow$CONCAT answer degradation (Sec. 5.3), retrieved-evidence recall (Sec. 5.4), recovery under RECAP and SNOWBALL (App. B.1), and the evidence-swap experiment in W2. Across these experiments, the same three retrievers consistently stand out as more robust to the conversational setting. This is why we believe query preprocessing is an important explanation for this robustness, although we concede that our experiments do not isolate it as the sole cause.

W2: Separating the two failure modes

The reviewer raises an important point about the distinction between locating where a gap appears and isolating its underlying cause. Our original FULL$\rightarrow$CONCAT$\rightarrow$SHARDED decomposition was intended primarily for the former. It shows whether degradation first appears when the original question is reformulated or when the same clues are subsequently distributed across turns. We agree, however, neither comparison isolates a single factor. We address their concerns below with additional experiments.

Isolating the Lost in Translation effect

Separating wording from retrieved evidence. We first isolate the roles of question wording and retrieved evidence by crossing the FULL and CONCAT questions with the evidence retrieved under the other condition. This gives the LLM either the FULL question with CONCAT evidence or the CONCAT question with FULL evidence.

RetrieverFULLCONCATFULL question + CONCAT evidence (FQCE)CONCAT question + FULL evidence (CQFE)
BM2539.830.130.139.8
Dense48.142.343.646.4
RAPTOR49.042.944.146.9
HippoRAG49.147.950.347.0
HippoRAG253.247.049.050.3
LightRAG39.939.039.939.5
ToG-249.246.948.347.0
All retrievers46.942.343.645.3

The table reports BEST F1 $\times$100, averaging Gemma-4-31B, Qwen3-14B, and Luna 5.6 over all five datasets.

We observe that most of the average FULL$\rightarrow$CONCAT loss comes from differences in the retrieved evidence. Changing the retrieved evidence produces a 3.0–3.3 F1 effect on average (CQFE−CONCAT, FULL−FQCE), while changing only the question wording produces a smaller 1.3–1.6 point effect (FQCE−CONCAT, FULL−CQFE). The same three retrievers that were robust to formulation in Sec. 5.3 (HippoRAG, LightRAG, and ToG-2) are also robust in this experiment.

Testing query rewriting. The reviewer asks whether the formulation loss could simply reflect the absence of query rewriting. We test this out using the mtRAG rewriting prompt over the visible conversation history with GPT-OSS 20B, the strongest rewriter identified by MTRAG-UN [7]. We evaluate Qwen3-32B across all five datasets and six retrievers, changing only the rewritten query passed to the retriever.

RetrieverFULLHISTORYCURRENTRewrite
BM2533.025.928.128.3
Dense43.535.230.031.5
RAPTOR44.837.130.732.3
HippoRAG42.938.534.134.3
HippoRAG250.340.736.436.3
LightRAG33.232.928.329.6
All retrievers41.335.131.332.1

We find that rewriting helps, but only modestly. It improves the incomplete CURRENT query, while direct HISTORY remains substantially stronger. This shows a direct contrast with mtRAG [6]. In its episodic, turn-level setting, rewriting is the strongest retrieval strategy, and CURRENT performs better than HISTORY among the raw conversational inputs. In our setting, no individual turn contains the complete information need. Consequently, the rewriter must go a step further and resolve ambiguities across several incomplete clues. Resolving even one entity incorrectly at any point in the conversation can contaminate the retrieval query, whereas HISTORY preserves the original clues.

W2 (continued)

Isolating the Lost in Conversation effect

Stricter paired complete-coverage analysis. The reviewer notes that the paired complete-coverage analysis in App. B.4 pools passages across turns, even though earlier passages are not carried into later contexts. We therefore repeat the analysis using a stricter criterion that retains only cases where the final SHARDED turn contains every supporting passage.

SHARDED Recall@5PairsQuestionsFULLSHARDED
100%52,60144673.862.8

The degradation remains essentially unchanged even when the retrieval has already supplied all required supporting evidence at the final turn.

Retaining previously retrieved passages. The reviewer also points to whether the degradation could result from not retaining passages retrieved in earlier turns. Our original protocol retrieves a fresh context at every turn, following the retrieve-then-read setting used in conversational RAG benchmarks [6–9]. Nevertheless, retaining earlier passages is a plausible alternative explanation, so we tested it directly by appending all previously retrieved passages to the current context.

LLMFULLHISTORY SHARDEDHISTORY Retain
Luna 5.663.757.557.9
Qwen3-14B49.540.439.5
Qwen3-32B51.746.046.4
All three LLMs55.048.047.9

Retaining all previously retrieved passages leaves HISTORY performance essentially unchanged. Earlier-passage eviction therefore does not explain the degradation.

Quantifying failure causes through manual review. We also investigate how LLM fails by manually reviewing 200 failed SHARDED conversations in which every supporting passage had been retrieved. We find that, in 18% of the conversations, the required passages arrived in different turns and were never available together, which is the case most directly addressed by retaining earlier evidence. Beyond this, the failures were heterogeneous. In 32%, the LLM lost track of the original intent, typically answering only the latest clue or stopping one hop short. It misread a retrieved passage in 16%, interpreted the underspecified request differently in 12%, and had sufficient evidence but never committed to the answer in 7%.

We observe more than one of these failures in 42% of the conversations. This helps explain why the progressively specified setting remains difficult even after the required evidence has been retrieved. To answer correctly, an LLM has to preserve the user’s original goal as new constraints arrive, resolve ambiguities in both the request and retrieved evidence, and connect information revealed at different turns.

We thank the reviewer for prompting us to examine the causes of degradation more carefully.

References

[1] Kwan et al. MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models.

[2] Han et al. Can Language Models Follow Multiple Turns of Entangled Instructions?

[3] Qian et al. UserBench: An Interactive Gym Environment for User-Centric Agents.

[4] Laban et al. LLMs Get Lost in Multi-Turn Conversation.

[5] Chen et al. KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues.

[6] Katsis et al. mtRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems.

[7] Rosenthal et al. MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations.

[8] Ali et al. RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark.

[9] Cheng et al. CORAL: Benchmarking Multi-Turn Conversational Retrieval-Augmented Generation.

We are writing to kindly ask whether our rebuttal has addressed your concerns. Please let us know if any concerns remain.

Reviewer_zd2ZOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies RAG systems in multi-turn conversations, where the evaluation data are formulated from multi-hop questions. The authors extends the framework proposed by Laban et al. [1] to RAG to examine the effects of question reformulation and multi-turn delivery. The experimental results reveal that multiturn interaction causes widespread performance degradation, making RAG systems simultaneously less accurate and less reliable. Moreover, this paper finds that there are two distinct failure modes behind this degradation, lost in translation or conversation.

[1] Laban, Philippe, et al. "Llms get lost in multi-turn conversation." International Conference on Learning Representations. Vol. 2026. 2026.

Critical strengths

The experiments are comprehensive, covering multiple LLMs, retrieval methods, and additional analyses.

Critical weaknesses
  1. Existing work has already studied RAG in multi-turn conversations, but the paper does not sufficiently discuss how its findings differ from or relate to prior work. Some relevant studies, such as KnowMT-Bench [1], which evaluates different RAG strategies in multi-turn question answering, are also missing from the discussion.

  2. The conversations are constructed by gradually revealing clues to a single question, which does not represent multi-turn QA in general. It is unclear whether the findings generalize beyond this specific setup.

[1] Chen, Junhao, et al. "Knowmt-bench: Benchmarking knowledge-intensive long-form question answering in multi-turn dialogues." arXiv preprint arXiv:2509.21856 (2025).

Questions for authors

Please see the weakness

3 comments

We sincerely appreciate the reviewer for taking the time to review our work and for recognizing the breadth of our experiments. We clarify the scope of our benchmark and its relationship to prior multi-turn work below.

W2: Scope and representativeness of the benchmark

We agree that progressively revealing clues to one question does not represent every form of multi-turn QA. Conversations may also contain topic shifts, independent follow-up questions, or changing user goals [4–10]. Our benchmark targets a different and practically important setting: the user has one fixed information need but initially expresses it incompletely, then supplies constraints that progressively identify the intended answer.

This setting is motivated by real user interactions, where requests are often ambiguous, underspecified, and revealed incrementally [1,2,11]. MARCH [3] further reports that ambiguous single-hop and multi-hop questions together account for 61.7% of sampled LMSYS-Chat-1M questions. This interaction between ambiguity and retrieval remains comparatively underexplored.

The ambiguity can originate in the user’s partial description or emerge when retrieval exposes several entities that match that description [12,13]. In our setting, an early ambiguity can redirect retrieval toward a different entity or supporting passage, changing the evidence available later in the conversation and making this a particularly challenging setting to study. The system must therefore preserve the original goal, avoid committing prematurely to a plausible but incorrect entity, and revise its interpretation as later clues arrive. As far as we know, ours is the first benchmark to systematically test retrievers and LLMs in this setting.

References

[1] Herlihy et al. On Overcoming Miscalibrated Conversational Priors in LLM-based Chatbots.

[2] Qian et al. UserBench: An Interactive Gym Environment for User-Centric Agents.

[3] Park et al. MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference.

[4] Chen et al. KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues.

[5] Katsis et al. mtRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems.

[6] Rosenthal et al. MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations.

[7] Ali et al. RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark.

[8] Cheng et al. CORAL: Benchmarking Multi-Turn Conversational Retrieval-Augmented Generation.

[9] Kwan et al. MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models.

[10] Han et al. Can Language Models Follow Multiple Turns of Entangled Instructions?

[11] Liu et al. We're Afraid Language Models Aren't Modeling Ambiguity.

[12] Min et al. AmbigQA: Answering Ambiguous Open-Domain Questions.

[13] He et al. A Small Model Asks Better: Fine-Tuning Query Clarification for Retrieval-Augmented Generation.

W1: Relationship to prior work

We next address how our findings differ from prior multi-turn work. We organize the closest literature into two related categories and explain how our experimental question and findings differ from each.

Conversational RAG Benchmarks

Existing conversational RAG benchmarks mostly study retrieval over episodic, turn-level information needs, each with its own answer or set of relevant passages [3,4,6,7]. KnowMT-Bench, for example, compares a fully-specified final question with the same complete question asked after several related conversational turns [3]. In other words, it compares related history + a fully specified final question against the same fully specified question alone. It finds that per-round RAG can reverse the multi-turn degradation in this setting. Other benchmarks study how a turn-level question can be resolved, rewritten, or retrieved using the preceding context [4,6,7].

Our setting differs because the turns are not separate questions, but constraints that jointly specify one information need.

When each turn poses an information need of its own, retrieving from the latest turn or rewriting it as a standalone query can be effective [4,5]. This helps explain why mtRAG finds the latest user turn to be the strongest raw conversational input and reports further gains from rewriting [4]. We therefore test whether query rewriting helps in our progressively specified setting using the mtRAG rewriting prompt and the strongest rewriter reported by MTRAG-UN [5]. The rewriter receives the conversation history and rewrites the current query to resolve ambiguities before retrieval.

We evaluate this with Qwen3-32B across all five datasets and six retrievers.

Retrieval inputF1Last-turn Recall@5
FULL41.357.2
SHARDED — HISTORY35.151.5
SHARDED — Rewrite32.144.1
SHARDED — CURRENT31.342.7

We find that rewriting improves over CURRENT but remains below HISTORY. In a progressively underspecified setting like ours, using the accumulated user turns as the retrieval query performs better.

Multi-Turn Interaction and Underspecification

A second line of work studies LLM behavior when information or instructions are distributed across interaction [1,2,8,9]. These studies show that LLMs can struggle to retain earlier content, reconcile related instructions, elicit missing information, or solve tasks revealed progressively. Their findings are largely orthogonal to ours because they do not examine retrieval or how the timing of the user's clues changes the external evidence returned by a RAG system.

Conversational RAG presents a broad space of design choices, including how the retrieval query is formed, which retriever is used, and how the LLM receives and uses the resulting evidence. Systematically identifying which of these choices remain robust when information is revealed progressively is itself a meaningful research contribution. Moreover, the conversational RAG benchmarks discussed above primarily evaluate lexical/sparse, dense, or conversationally fine-tuned retrievers [3–7], whereas our evaluation also includes modern graph-based multi-hop retrieval baselines.

We will revise Secs. 2–3 to make this scope and comparison with prior work clearer, and add the new query-rewriting analysis to the revised Results and appendix.

References

[1] Qian et al. UserBench: An Interactive Gym Environment for User-Centric Agents.

[2] Laban et al. LLMs Get Lost in Multi-Turn Conversation.

[3] Chen et al. KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues.

[4] Katsis et al. mtRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems.

[5] Rosenthal et al. MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations.

[6] Ali et al. RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark.

[7] Cheng et al. CORAL: Benchmarking Multi-Turn Conversational Retrieval-Augmented Generation.

[8] Kwan et al. MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models.

[9] Han et al. Can Language Models Follow Multiple Turns of Entangled Instructions?

Thanks for the clarification. The response largely addresses my concerns about the relationship to prior work and the scope of the findings. I will adjust my score accordingly.

Reviewer_5wZ3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on the performance of RAG systems in multi-turn dialogues. Specifically, it transforms questions from multi-hop question answering datasets into multi-turn dialogues to measure the performance loss of RAG systems when the same question is presented in multiple turns. The authors first use an LLM to break down each question into an opening turn stating what the user is looking for and several clues. And then they use this generated data to simulate clue-by-clue retrieval and question answering. The authors compare three conditions: the original question is presented all at once, all clues are presented in the same turn, and clues are presented in multiple turns. The authors run the experiments with multiple LLMs and retrieval methods. The results show that the performance of all retrieval methods degrades in multi-turn dialogues. The authors also analyze the reasons for the performance degradation of different retrieval methods.

Critical strengths

I think the one critical strength of this paper is its simple and clear experimental design. Specifically, in the authors' experiments, each time the same question is presented to the model in three different ways. This prevents the influence of varying question difficulty and allows the analysis to focus on the presentation of the information itself. I also appreciate the breadth of the experiments. The paper tested multiple retrieval methods and LLMs.

Critical weaknesses

I think one weakness of this paper is the splitting and ordering of clues. All the clue splitting and ordering in the paper are generated by a relatively old model, GPT-4o-mini. However, in multi-round input conditions, the way clues are split and their order directly affects the difficulty of the model's answer. Clues for multi-hop questions should ideally start with directly searchable entities and then gradually work towards the final answer. For example, in Table 5, the question about the author of the Kubrick movie only reveals the specific movie's album information in the third round and the important relationship is only given in the final round. This makes it hard to distinguish how much of the performance degradation is due to multi-round input and how much is due to unreasonable clue splitting and ordering. Another weakness is the way the authors present the retrieved documents. In the multi-round setting, the model can only see the documents retrieved in the current round. Documents retrieved in previous rounds are not retained in the context. If only the current round's retrieval is used, the final query may struggle to find useful documents. Even if relevant documents were found in the earlier rounds, they are no longer in the context of the final round. The paper uses the combined results from each round to demonstrate that evidence has been retrieved, thus saying that the model failed to integrate cross-round evidence. Since the model never simultaneously sees these documents, I believe this conclusion requires further evidence.

Questions for authors

Can you provide more details about the results of the human evaluation of the GPT-4o-mini model?

3 comments

We thank the reviewer for their careful reading and constructive feedback. We are delighted that they found the experimental design clear and appreciated the breadth of the evaluation. Their concerns about shard construction and evidence retention motivated several additional experiments that, we believe, substantially strengthen the analysis. We address them below.

All tables report BEST F1 $\times$100.

W1. Splitting and ordering of clues

Sharding with a newer LLM model: Our choice of GPT-4o-mini was informed by Laban et al. [1] who identified it as a reliable, low-cost LLM for sharding. To test the impact of a newer model, we randomly sampled 20 questions from each of our five datasets and regenerated all 100 shard sets using Luna 6 with the same construction prompts.

We then reran CONCAT and SHARDED using Luna 5.6 and GPT-4o-mini as the LLM assistants, with closed book, BM25, Dense, and HippoRAG2.

FULLCONCAT: GPT-4o-miniCONCAT: Luna 6SHARDED: GPT-4o-miniSHARDED: Luna 6
48.141.841.742.541.5

The results indicate that sharding with Luna 6 does not alter the paper's conclusions, as FULL outperforms SHARDED regardless of the sharding model.

Impact of clue ordering. Similar to Laban et al., each conversation starts with an intent shard defining the target (e.g., person, place, or date) while leaving identifying constraints open. We do not enforce the first shard to contain directly searchable entities, so some of the shards may or may not have them. However, this does not leave the retriever without those entities, as they appear in later clues and, under HISTORY, end up in the accumulated retrieval query.

To measure the impact of their order, we manually reordered 50 questions that admitted another valid ordering without changing any wording. We kept the intent shard first and excluded questions whose clues formed a strict dependency chain with no other valid order.

LLMPolicyFULLOriginal orderReordered
Luna 5.6HISTORY71.763.263.6
Luna 5.6CURRENT71.765.063.7
Qwen3-14BHISTORY54.946.544.8
Qwen3-14BCURRENT54.943.941.2
Qwen3-32BHISTORY56.652.349.7
Qwen3-32BCURRENT56.648.047.4

These findings suggest that alternative clue ordering does not primarily drive the observed degradation.

W2. Retaining evidence from previous turns

Our setup follows prior conversational RAG benchmarks that compose a fresh context from information retrieved at each turn [2–4]. However, we agree that retaining evidence from previous turns may help the LLM stay informed in subsequent rounds.

To address the reviewer's concern, we reran SHARDED while retaining all previously retrieved passages. We evaluate this across three LLM assistants, six retrievers, and both query policies.

LLMFULLHISTORY: SHARDEDHISTORY: RetainCURRENT: SHARDEDCURRENT: Retain
Luna 5.663.757.557.956.857.9
Qwen3-14B49.540.439.538.839.3
Qwen3-32B51.746.046.442.945.1
All three LLMs55.048.047.946.247.4

We observed that retention does not fix the FULL–SHARDED gap. Under HISTORY, where prior clues already remain available to the retriever, retaining passages provides essentially no additional benefit. Under CURRENT, it recovers a small amount of performance by preserving evidence that the current-turn query can no longer retrieve.

While retaining information can be beneficial in certain settings, a growing context filled with duplicate or irrelevant evidence may degrade both scalability and answer accuracy in long conversations.

References

[1] Laban et al. (2026). LLMs Get Lost in Multi-Turn Conversation. ICLR 2026.

[2] Katsis et al. (2025). mtRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems. TACL 13.

[3] Rosenthal et al. (2026). MTRAG-UN: A Benchmark for Open Challenges in Multi-Turn RAG Conversations. Findings of ACL 2026.

[4] Cheng et al. (2025). CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation. Findings of NAACL 2025.

Q. Human validation of the GPT-4o-mini shards

We manually reviewed all 750 shard sets before running any benchmark experiments, spending approximately 5–10 minutes on each question. Each review checked whether the shards satisfied the six construction rules in Appendix A.1, and we corrected any violations by directly editing the problematic shards before running the benchmark.

For additional verification, following prior benchmark practice [1–4], we asked a separate LLM, Luna 6, to compare every reviewed shard set with its original FULL question using a three-point rubric.

RatingMeaningQuestions
3The shards preserve the complete question669 (89.2%)
2The shards preserve what is being asked, but omit, add, or alter a clue81 (10.8%)
1The shards change what is being asked0

The average rating is 2.892. Most score-2 cases come from MuSiQue 3-hop and 4-hop, where the original questions are themselves more ambiguous, making it harder for the validator to identify the intended entities and relations.

References

[1] Ali et al. (2026). RECOR: Reasoning-focused Multi-turn Conversational Retrieval Benchmark. Findings of ACL 2026.

[2] Park et al. (2026). MARCH: Evaluating the Intersection of Ambiguity Interpretation and Multi-hop Inference. Findings of ACL 2026.

[3] Tang and Yang (2024). MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries. COLM 2024.

[4] Su et al. (2025). BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval. ICLR 2025.

Thanks for the reply and the experiments. They addressed most of my concerns. In particular, I appreciate the one on retaining passages from previous turns. I plan to keep my score since it is already a positive score.

Comments on the paper

1 comment

Authors' Final Remarks

We sincerely thank all reviewers for their careful feedback and for helping us strengthen both the analysis and presentation of our work. We briefly summarize the contributions of our paper and explain how we addressed reviewers' concerns.

Our paper introduces a benchmark for RAG systems in which one underspecified, multi-hop information need is progressively revealed across turns. RAG encompasses a broad design space spanning the query supplied to retrieval, the retrieval architecture itself, the evidence retained across turns, and the ability of the LLM to interpret and combine that evidence. We systematically evaluate these parts of the pipeline across different retrievers, LLMs, datasets, and query policies. We also test both our proposed mitigations and alternatives suggested by the reviewers, to produce findings that can guide researchers and practitioners as they develop conversational RAG systems.

During the rebuttal, we addressed the following central concerns:

  • Retaining previously retrieved passages does not resolve multi-turn degradation.
  • Reordering the clues produces no meaningful change in performance, ruling out clue ordering as a source of the observed degradation.
  • We further isolated the Lost in Translation and Lost in Conversation effects with additional ablations.
  • Conversational query rewriting does not resolve multi-turn degradation in our highly ambiguous setting and performs worse than the simpler HISTORY query.
  • We clarified the scope and novelty of our work relative to prior research. Our turns jointly specify one question rather than posing episodic, turn-level information needs like complementary multi-turn RAG benchmarks.

We are glad to have addressed Reviewer zd2Z's concerns about prior work and the scope of our findings.

We believe that enabling LLMs and retrieval systems to jointly resolve ambiguity, without allowing an early mistaken interpretation to contaminate subsequent retrieval and generation, is an important next step for conversational RAG. We hope our benchmark provides a bridge between idealized standalone multi-hop evaluation and the progressively specified requests that arise in real conversations.

Irrespective of the outcome of the paper, we will incorporate the new experiments and expand the related work in the future revision of the paper. We sincerely thank the reviewers again for their time and constructive feedback.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.