- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies why AI sometimes gives wrong answers even when it has access to long conversation histories. The authors find that too much information can confuse AI, while retrieving only relevant messages may miss important details. They propose that AI should act like a detective: not just find relevant information, but connect different clues and check whether they are enough to support the answer. However, this idea has not yet been fully proven.
Critical strengths
-
The paper studies an important problem in long-dialogue question answering. Finding relevant information does not always mean finding enough information to answer a question correctly. This problem is worth further research.
-
The authors manually analyze 24 questions in addition to their quantitative experiments. This provides another way to understand the results and helps identify problems with the gold evidence in existing datasets.
Critical weaknesses
-
The paper is difficult to follow, and many important ideas and methods are not clearly explained. The Introduction only cites one study from 2024 and does not provide enough discussion of recent related work. As a result, it is difficult to understand how this paper differs from existing studies and what its main contribution is.
-
The paper proposes the Semantic-Chain idea, but the main experiments use existing retrieval methods, such as BM25, Dense Retrieval, RRF, and Reranker. Although the authors manually analyze some evidence chains, they do not show whether using Semantic Chains can actually improve the final answer accuracy. This makes it difficult to judge the usefulness of the proposed idea.
-
In Section 2, the authors increase the context length, add more distracting information, and change the position of relevant evidence at the same time. Therefore, it is unclear what actually causes the performance drop. The experiment also uses only 20 questions, and repeated runs show large differences (e.g., 17/20 vs. 12/20 at around 606K tokens). Moreover, only one model with low reasoning effort is tested, so it is unclear whether the findings apply to other models or settings.
-
The paper suggests that a complete answer can be supported by a chain of connected evidence from the original dialogue. However, the authors do not provide a complete mathematical proof, and the manual analysis covers only 24 questions. More theoretical or experimental evidence is needed to support this general claim.
Questions for authors
-
How is your work different from existing studies on long-term memory and evidence retrieval? What is the main new contribution of this paper?
-
Can you provide experiments showing that the proposed Semantic-Chain approach actually improves question-answering accuracy compared to existing retrieval methods?
-
In Section 2, context length, distracting information, and evidence position change at the same time. How can you tell which factor causes the performance drop? Would the results remain similar with other models or higher reasoning effort?
-
What evidence supports the claim that a complete answer can always be supported by a linear evidence chain? Is the manual analysis of 24 questions enough to support such a general hypothesis?
2 comments
Please engage with the authors during the discussion period.
Dear reviewer,
The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale. (I realize that there is only one business day between when the authors posted their response and when I sent this reminder -- please read it as just a friendly reminder and not an implication that I think you're being unresponsive).
Q1. What is the main contribution beyond existing work on long-term memory and evidence retrieval?
Our contribution is not a new way to rank dialogue turns. We propose making explicit the relations that must hold for an answer to follow, so that a reader can check each relation against the source. For example, case 10 in the supplementary audit asks:
“Which batch of sweet potatoes did I incorporate into breakfast preparations on the morning Jasper curated the playlist and set the table, considering my earlier produce purchases?”
The question requires two links: the breakfast must be the one prepared on the morning of Jasper’s activities, and the batch used for that breakfast must be the one identified in an earlier purchase. Retrieving a purchase and a breakfast does not, by itself, establish that they involved the same batch. A chain makes it possible to ask precisely which passage supports that identification. It can also guide the next search step: once the breakfast is identified, what remains to be found is evidence of the batch’s origin, not another mention of breakfast.
Q2. Do the experiments show that semantic chains improve final answer accuracy?
No. In Table 5, we deliberately stopped at retrieval. For each question, we submitted two queries to the same vector search system: the text of the chain and the text of the Gold Evidence turns. Both queries ranked the same dialogue chunks using the same index. We then compared the ranks assigned to the same set of evidence chunks under each query.
Passing the retrieved chunks to an answer model would introduce another factor: even when the necessary evidence is retrieved, the model can fail to use it correctly. Our narrower question was how the ranking changes when the query changes from Gold Evidence text to chain text while the index and retrieval method remain fixed. Table 5 addresses that question; it does not establish an improvement in final answer accuracy or isolate the effect of chain structure from query content.
Q3. Which factor caused the performance drop, and does it generalize to other models or reasoning settings?
The experiment in Section 2 does not distinguish those causes. We kept the same 20 questions and measured what happened as the dialogue history expanded. Context length, the amount of distracting material, and evidence position changed together; we did not vary them independently. Even at approximately 60K tokens, two runs yielded 17/20 and 12/20 correct answers. We therefore interpret the result as a description of performance on this panel under the tested configuration, not as a universal context-length threshold or evidence that length alone caused the drop. We did not test other reader models or higher reasoning effort in this experiment.
Q4. What supports the chain-existence claim, and are 24 audited questions sufficient?
The 24 cases do not prove a universal claim. The original submission presented chain existence as a hypothesis. The supplementary document Semantic_Chains.pdf provides a formal existence proof under the assumptions stated in the theorem. The formulation permits facts to be grouped into a single link where appropriate; it does not require every valid chain to contain multiple links. Existence is also distinct from discovery: the theorem does not guarantee that the proposed search procedure will find a valid chain.
The empirical audit in Dialogue_Evidence_Audit_24_Questions.pdf serves a different purpose. It shows which relations can be verified in actual dialogues, which require metadata, and where a premise or part of an answer remains unsupported. These examples illustrate how to apply the framework and where verification stops; they are not the basis for the theorem’s generality.