acceptodds
Under review as a conference paper at ICLR 2027

From Relevant Fragments to Sufficient Answers: Verifiable Evidence Chains in Long Dialogues

Abstract

Long-context language models can process large dialogue histories, yet supplying more context does not reliably improve memory-based question answering. We study this through a fixed-panel reader experiment at increasing input lengths and through 50K-token, 100K-token, full-history, and oracle-evidence conditions on three long-dialogue benchmarks. Performance is non-monotonic and dataset-dependent: 50K packages are close to Full on RHELM and LoCoMo Long but substantially worse on EverMemBench, and coverage of Gold Evidence does not guarantee answer correctness. These gaps motivate a hypothesis of evidence sufficiency: when a complete answer is present in a finite textual context, at least one linear chain of source facts, events, and objects joined by verifiable lexical keys supports it. A manual audit of 24 blindly selected questions illustrates chain reconstruction and exposes limitations of official annotations, including dependencies on external metadata.

Primary area:
foundation or frontier models, including LLMs
Keywords:
long-context language models, dialogue memory, evidence retrieval, retrieval-augmented generation, context compression, multi-hop reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_X5qYOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies why AI sometimes gives wrong answers even when it has access to long conversation histories. The authors find that too much information can confuse AI, while retrieving only relevant messages may miss important details. They propose that AI should act like a detective: not just find relevant information, but connect different clues and check whether they are enough to support the answer. However, this idea has not yet been fully proven.

Critical strengths
  1. The paper studies an important problem in long-dialogue question answering. Finding relevant information does not always mean finding enough information to answer a question correctly. This problem is worth further research.

  2. The authors manually analyze 24 questions in addition to their quantitative experiments. This provides another way to understand the results and helps identify problems with the gold evidence in existing datasets.

Critical weaknesses
  1. The paper is difficult to follow, and many important ideas and methods are not clearly explained. The Introduction only cites one study from 2024 and does not provide enough discussion of recent related work. As a result, it is difficult to understand how this paper differs from existing studies and what its main contribution is.

  2. The paper proposes the Semantic-Chain idea, but the main experiments use existing retrieval methods, such as BM25, Dense Retrieval, RRF, and Reranker. Although the authors manually analyze some evidence chains, they do not show whether using Semantic Chains can actually improve the final answer accuracy. This makes it difficult to judge the usefulness of the proposed idea.

  3. In Section 2, the authors increase the context length, add more distracting information, and change the position of relevant evidence at the same time. Therefore, it is unclear what actually causes the performance drop. The experiment also uses only 20 questions, and repeated runs show large differences (e.g., 17/20 vs. 12/20 at around 606K tokens). Moreover, only one model with low reasoning effort is tested, so it is unclear whether the findings apply to other models or settings.

  4. The paper suggests that a complete answer can be supported by a chain of connected evidence from the original dialogue. However, the authors do not provide a complete mathematical proof, and the manual analysis covers only 24 questions. More theoretical or experimental evidence is needed to support this general claim.

Questions for authors
  1. How is your work different from existing studies on long-term memory and evidence retrieval? What is the main new contribution of this paper?

  2. Can you provide experiments showing that the proposed Semantic-Chain approach actually improves question-answering accuracy compared to existing retrieval methods?

  3. In Section 2, context length, distracting information, and evidence position change at the same time. How can you tell which factor causes the performance drop? Would the results remain similar with other models or higher reasoning effort?

  4. What evidence supports the claim that a complete answer can always be supported by a linear evidence chain? Is the manual analysis of 24 questions enough to support such a general hypothesis?

2 comments

Q1. What is the main contribution beyond existing work on long-term memory and evidence retrieval?

Our contribution is not a new way to rank dialogue turns. We propose making explicit the relations that must hold for an answer to follow, so that a reader can check each relation against the source. For example, case 10 in the supplementary audit asks:

“Which batch of sweet potatoes did I incorporate into breakfast preparations on the morning Jasper curated the playlist and set the table, considering my earlier produce purchases?”

The question requires two links: the breakfast must be the one prepared on the morning of Jasper’s activities, and the batch used for that breakfast must be the one identified in an earlier purchase. Retrieving a purchase and a breakfast does not, by itself, establish that they involved the same batch. A chain makes it possible to ask precisely which passage supports that identification. It can also guide the next search step: once the breakfast is identified, what remains to be found is evidence of the batch’s origin, not another mention of breakfast.

Q2. Do the experiments show that semantic chains improve final answer accuracy?

No. In Table 5, we deliberately stopped at retrieval. For each question, we submitted two queries to the same vector search system: the text of the chain and the text of the Gold Evidence turns. Both queries ranked the same dialogue chunks using the same index. We then compared the ranks assigned to the same set of evidence chunks under each query.

Passing the retrieved chunks to an answer model would introduce another factor: even when the necessary evidence is retrieved, the model can fail to use it correctly. Our narrower question was how the ranking changes when the query changes from Gold Evidence text to chain text while the index and retrieval method remain fixed. Table 5 addresses that question; it does not establish an improvement in final answer accuracy or isolate the effect of chain structure from query content.

Q3. Which factor caused the performance drop, and does it generalize to other models or reasoning settings?

The experiment in Section 2 does not distinguish those causes. We kept the same 20 questions and measured what happened as the dialogue history expanded. Context length, the amount of distracting material, and evidence position changed together; we did not vary them independently. Even at approximately 60K tokens, two runs yielded 17/20 and 12/20 correct answers. We therefore interpret the result as a description of performance on this panel under the tested configuration, not as a universal context-length threshold or evidence that length alone caused the drop. We did not test other reader models or higher reasoning effort in this experiment.

Q4. What supports the chain-existence claim, and are 24 audited questions sufficient?

The 24 cases do not prove a universal claim. The original submission presented chain existence as a hypothesis. The supplementary document Semantic_Chains.pdf provides a formal existence proof under the assumptions stated in the theorem. The formulation permits facts to be grouped into a single link where appropriate; it does not require every valid chain to contain multiple links. Existence is also distinct from discovery: the theorem does not guarantee that the proposed search procedure will find a valid chain.

The empirical audit in Dialogue_Evidence_Audit_24_Questions.pdf serves a different purpose. It shows which relations can be verified in actual dialogues, which require metadata, and where a premise or part of an answer remains unsupported. These examples illustrate how to apply the framework and where verification stops; they are not the basis for the theorem’s generality.

Area_Chair_ZPpuOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale. (I realize that there is only one business day between when the authors posted their response and when I sent this reminder -- please read it as just a friendly reminder and not an implication that I think you're being unresponsive).

Reviewer_HzZNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper combines a fixed-panel reader experiment with comparisons of 50K, 100K, Full, and Oracle reader-input conditions on three long-dialogue benchmarks, proposes a semantic-chain hypothesis connecting questions to sufficient answers through source facts and lexically expressed transition keys, and uses a 24-question source-locked audit to illustrate chain reconstruction and annotation limitations.

Critical strengths
  1. The design is cumulative: the effective-window scan on a fixed panel motivates the reduced-context comparison, whose dataset-dependent gaps and coverage–accuracy mismatches motivate the sufficiency question.

  2. The hypothesis is motivated by an explicit argument rather than merely asserted: from the lexical-key formulation through finite reference resolution to link-by-link verification, with universal chain existence explicitly left unproven.

  3. The reported audit protocol is disciplined: questions were selected blindly, chains were constructed with BM25 from question-derived keys and expanded through keys in retrieved assertions, Gold Evidence was inspected only after the approved chain query had ranked the source chunks, and post-inspection corrections were saved as separate versions.

Critical weaknesses
  1. The semantic-chain hypothesis remains insufficiently operationalized. Although the paper defines Full(C,Q,A)\mathrm{Full}(C,Q,A) independently of the chain and provides qualitative chain-checking rules, it does not spell out a reproducible protocol for making that independent sufficiency judgment or explain how an unsuccessful chain search would be distinguished from a genuine counterexample to universal chain existence. Compound-fragment admissibility is insufficiently formalized, and the completeness measure Φ\Phi lacks a concrete numerical specification. The paper also does not clearly position or validate the incremental contribution of its proposed linear-chain structure relative to prior sufficient-context analysis (Sufficient Context[1]).

  2. The evaluation does not isolate the contribution of chain structure. Tables 2–4 test retrieval and context packing, not chain construction. Table 5’s Mean Dense rank comparison holds the index and evaluated chunk set fixed but does not control for query content or length; downstream answer accuracy for chain-derived inputs is not reported. The 24-question audit is author-adjudicated, with no reported independent inter-annotator agreement.

  3. The evidence is limited in scale. The fixed-panel reader study uses only 20 questions with two to three independent runs per reported length. The systematically reported empirical evidence for chain reconstruction is limited to a 24-question manual audit.

[1] Joren H, Zhang J, Ferng C S, et al. Sufficient context: A new lens on retrieval augmented generation systems. International Conference on Learning Representations. 2025, 2025: 20310-20334.

Questions for authors
  1. What testable implication or reproducible verification benefit do lexical-key chains add beyond Sufficient Context[1], and what evidence supports that increment?

  2. Given the stated pre-Gold construction protocol, please confirm which chain versions produced Table 5 and identify any post-inspection revisions used. Is there a comparison using the same facts and matched query lengths that isolates chain structure from content?

  3. Please report independent inter-annotator agreement and counts of text-answerable, chain-successful, unresolved, and metadata-dependent cases. Among questions independently judged answerable from CC alone, how often was a valid chain constructed, and what empirical support does this audit provide for universal chain existence?

2 comments

Q1. What do lexical-key chains add beyond a sufficient-context judgment?

A sufficient-context judgment asks whether the available context contains enough information to answer the question. A chain makes explicit which relations must hold for the answer to follow and which source passages support each relation. This gives a reader a specific way to check the answer: they can challenge an individual link rather than accept or reject the context as a whole.

For example, case 10 in the supplementary audit asks: “Which batch of sweet potatoes did I incorporate into breakfast preparations on the morning Jasper curated the playlist and set the table, considering my earlier produce purchases?” Two relations need to be established. The breakfast must be the one prepared on the same morning as Jasper’s activities, and the sweet potatoes used for that breakfast must come from the particular purchase identified in the answer. A passage about buying sweet potatoes and another about cooking them do not, by themselves, establish that they concern the same batch. The chain lets a reader check the timing and the batch identity separately. It can also guide further search: once the breakfast has been identified, the missing evidence concerns the origin of the batch, not breakfast in general. The audit illustrates this verification procedure; it does not establish that it improves retrieval or answer accuracy.

Q2. Which chain versions were used in Table 5, and was chain structure tested separately?

Before inspecting Gold Evidence, we submitted the full text of the constructed chain as the retrieval query. We did not reorder the links while holding the facts and query length fixed, because we did not separately evaluate the effect of chain structure on retrieval rank. After submission, we updated the rules for identifying links and revised the chains in the supplementary audit. We did not rerun the ranking experiment on the updated chains. Table 5 reports results for the original version of the chains.

Q3. How many cases were text-answerable, how were the chains checked, and what does the audit establish?

Under a conservative author assessment, 5 of the 24 original questions (cases 1, 4, 6, 7, and 8) could be fully answered from the ordered dialogue utterances alone, without speaker labels, session dates, or other metadata. In 2 of those 5 (cases 1 and 4), the complete chain as recorded was also supported by utterance text alone. In cases 6–8, the text sufficed for the requested answer, and speaker identity could be recovered from neighboring turns containing direct forms of address. The recorded chains, however, included calendar details beyond what the questions required. This distinction matters: those three cases were not failed attempts to construct a chain.

Across all 24 cases, 2 recorded justifications were fully supported by utterance text alone, 11 were fully supported with metadata, and 11 provided conditional or partial support. The last category does not mean that no answer could be identified. In case 14, for example, the route could be identified, but the dialogue did not establish the question’s premise that the speaker had already described the discovery to the group. In case 15, the requested frequency comparison could not be established.

The author checked the reconstructions against the source passages. The follow-up audit had access to the Gold annotations, and no second annotator independently assessed answerability or chain validity. We therefore cannot report inter-annotator agreement or a success rate on an independently judged text-answerable subset. The audit illustrates successful reconstructions and identifies where verification stops; it does not, by itself, establish universal chain existence.

I thank the authors for their response. However, my concerns about the weaknesses of this study have not been sufficiently addressed, and I therefore maintain my original rating.

Reviewer_SikQOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates evidence use in long-dialogue question answering. It evaluates 20 fixed questions at increasing context lengths, compares retrieved context with full histories and annotated evidence on three datasets with 100 questions each, and manually audits 24 questions. It proposes that every answer fully supported by a finite dialogue context admits a linear chain of source facts connected through verifiable lexical references. The audit reconstructs candidate chains and reports defects in questions, reference answers, and evidence annotations. Chain existence is presented as a hypothesis, not a proved result.

Critical strengths

S1. The manual audit provides concrete examples of potential annotation defects, which may help guide further inspection of the evaluated datasets.

Critical weaknesses

W1. The paper does not explain its central idea clearly enough. Evidence chains are the main proposed contribution, but the manuscript does not walk through a complete example from a question, through the original evidence and its connections, to an answer. Sections 5–6 introduce substantial notation, yet do not specify how the completeness function Phi is evaluated or show how it is used in the experiments. The brief editing example does not fill this gap. This makes it difficult to understand what the formalism contributes and to reproduce the assessment of a proposed chain.

W2. The experiments do not provide convincing support for the evidence-chain hypothesis. The first changes context length, distractors, and evidence position together, so the effects cannot be separated. The second does not distinguish retrieval failures from incorrect or incomplete Gold annotations, reader errors, or judge errors; using the same Gold annotations as Oracle inputs does not resolve this problem (Sections 2–4). The final audit covers only 24 questions and does not clearly report how many satisfy the text-only assumptions and yield a complete valid chain, or whether these judgments were independently checked. Table 5 reports retrieval ranks rather than a direct test of chain validity. These experiments leave the central hypothesis weakly supported.

W3. The paper does not establish what researchers learn beyond a familiar problem. The observation that relevant fragments may not contain enough evidence to answer a question is not, by itself, a substantial contribution, particularly given the supporting-evidence and multi-hop reasoning work discussed in Section 5. The paper needs to show what its chain formulation adds to that understanding. However, it neither demonstrates that chains improve answer quality nor establishes that they diagnose errors more reliably. The annotation audit also does not show how the reported defects affect the main evaluation results. As presented, the paper identifies a reasonable concern but does not establish a sufficiently clear new finding or a demonstrated way to address it.

Questions for authors

Q1. What does a linear evidence chain add beyond finding a sufficient set of facts? Please show one complete example, from the question through the source passages and their connections to the answer, and explain how you determine that the chain provides sufficient evidence.

Q2. In the second experiment, how do you distinguish retrieval failures from incorrect or incomplete Gold annotations, reader errors, and judge errors? The Oracle condition uses the same Gold annotations, so it does not resolve this ambiguity. Do the existing results show which of these causes accounts for the observed failures?

Q3. Of the 24 audited questions, how many can be answered from the dialogue text alone, and for how many of those did you construct a complete valid chain? How was validity checked, and was anyone other than the author involved in validating the chains? Please also report unresolved cases and explain why they could not be resolved.

3 comments

Q1. What does a linear chain add beyond a sufficient set of facts?

A chain does not add evidence to a set that is already sufficient. It makes explicit which source passages support each part of the answer and what licenses the transitions between them. A shared person or topic can connect two passages without establishing the particular relation the question asks about.

For example, LoCoMo conv-41-q099 asks what John hosted for veterans in May 2023 as part of his project. In D15:1, John identifies the project as supporting military veterans. In D15:11–12, he mentions a photo from “last Friday,” and Maria asks what he did; the session date anchors the exchange in May. In D15:13, John answers: “We had a great time throwing a small party and inviting some veterans to share their stories.” The answer can be expanded into two links: John and others hosted a small party → they invited veterans to share their stories. The common subject we establishes that the organizers and the people extending the invitation are the same group. The words to share their stories, not the shared subject alone, establish the purpose of the invitation. The session date is metadata; we do not count this as a text-only case.

Our sufficiency check was qualitative: for each part of the answer, we identified its source and checked whether the claimed relation was supported. We did not compute a numerical value of Φ. The theoretical result concerns the existence of a chain representation under stated assumptions, including cases where the answer is derived by an operation on supported facts. It does not guarantee that a retrieval procedure will find the chain or that presenting it will improve answer accuracy.

Q2. Can the second experiment distinguish retrieval, annotation, reader, and judge errors?

Not for each incorrect answer. We measured two different outcomes: coverage of the dataset’s annotated evidence and agreement of the generated answer with the dataset’s reference answer. The two judges saw the question, the reference answer, and the candidate answer, but not the source dialogue. Oracle bypasses automated retrieval by supplying the annotated evidence; it does not independently establish that those annotations are sufficient.

RHELM aggregation_9e30df illustrates why this matters. The question asks which sumac dish appeared most frequently at dinner. Both Gold turns concern a chickpea salad at one gathering, where it is described as a snack. They identify the dish but do not establish comparative dinner frequency. Here the audit identifies a specific limitation of the annotation; it does not establish why a particular model output received a negative judgment. Likewise, in the first experiment, the evidence-bearing sessions were retained, but we did not isolate the effects of input length, distractors, and evidence position.

Q3. How many of the 24 questions were answerable from text alone, and how were the chains checked?

In our retrospective audit, 5 of 24 questions were fully answerable from the ordered utterance text alone: cases 1, 4, 6, 7, and 8. We excluded speaker fields, timestamps, session dates, group identifiers, and attachment captions. Names and temporal expressions spoken in the dialogue counted as text. For when questions, we accepted a dialogue-relative answer such as “last Friday” when the question did not require a calendar date.

For 2 of those 5 (cases 1 and 4), the complete chain as written in the report can also be justified from utterance text alone. In cases 6–8, the recorded chain includes calendar details beyond what the question requires: the months of two hikes rather than just their count, or a session date in addition to the relative timing of an event. We checked speaker attribution in these cases against neighboring turns containing direct forms of address. The distinction is between what the question requires and what the recorded chain asserts, not three failed attempts to construct a chain.

The author checked the reconstructions against the source turns. The follow-up audit had access to Gold annotations; it was not an independent blind validation, and no second annotator assessed agreement. We also distinguish partial answers from unsupported premises. In case 15, the requested frequency comparison could not be established. In case 14, the route itself could be identified as the Kinzig Valley route, but the dialogue did not establish the question’s premise that the speaker had already recounted his discovery to the group.

Area_Chair_ZPpuOpenReview ↗

Please engage with the authors during the discussion period.

Dear reviewer,

The authors have provided a response to your review. Please ensure that you have read it and taken it into consideration when making your final recommendation. Even if you do not change your score, please post a response with your rationale. (I realize that there is only one business day between when the authors posted their response and when I sent this reminder -- please read it as just a friendly reminder and not an implication that I think you're being unresponsive).

Reviewer_SikQOpenReview ↗

Follow-up on Authors' Response

Thank you for your response. However, the clarifications do not resolve my core concerns regarding the experimental attribution, the lack of quantitative evaluation or downstream gains for the proposed evidence chains, and the limitations of the 24-question audit. Therefore, I will maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.