- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In a long conversation sessions with an agent, LLMs can produce continuations that directly contradict earlier claims or alternatively produce continuations that depend on earlier conversation that was abandoned. The paper focuses on producing continuations that are grounded in premises that are valid. The paper proposes to map utterances to one or more of eight epistemic operation (with an LLM), which is then applied to a dependency graph by a symbolic engine. Then checking from grounded-ness of any continuation is walk on this dependency graph. This guarantee depends on correctly mapping claims and their dependencies into the graph.
Critical strengths
- Simple Mechanism: The mechanism builds a dependency graph that can be checked against continuations in symbolic terms. This provides a simple and powerful mechanism which shows very good results
- Promising Empirical Results: Evaluation results show that tracking premise validity can allows LLMs to self-correct providing higher results at least in QA.
Critical weaknesses
- Unclear Framing: The abstract starts with "context-manipulation attacks" and then the intro follows up incorrect continuations which is a bizarre disconnect from the reader's perspective. It would be better for a consistent framing of the problem.
- Indirect Evaluation of the Central Claim: The evaluation is indirectly measuring the impact of tracking premise validity on performance of various tasks. However, it does not directly measure whether the verifier correctly accepts supported candidate continuations and rejects unsupported ones, particularly when claim to record links and dependencies must be inferred. Such an evaluation would better establish the scope and reliability of the continuation verification claim.
Questions for authors
- Could you evaluate candidate answer or continuation sampled from the QA tasks and independently label whether their claims are supported by the currently valid premises?
- Report how often unsupported continuations are rejected?
- Can you also report how often valid continuations are accepted?
4 comments
Follow-up to Reviewer WHpW
Dear Reviewer WHpW,
Thank you for the time and care you put into reviewing our paper.
Your comments changed the paper for the better. Your questions on how often valid continuations are accepted and unsupported ones are rejected led to the new Section 4.4, a direct evaluation of the grounding check. Your point on the framing led us to revise the abstract so that it presents the same problem as the introduction. Your question on inferred links also pushed us to test the case where an LLM performs the association (Appendix D.4).
If anything in our response or the revised paper is still unclear or unconvincing to you, we would be glad to hear it. Your feedback is valuable to us either way.
Thank you again.
Thanks for addressing my concerns
I thank the authors for addressing the concerns raised and doing the requested additional analysis.
The end-to-end arm in the table in Q1 is the case where the dependencies are inferred: 0.989–0.995 of the valid answers are accepted and 0.994–1.000 of the unsupported ones are rejected.
Such high numbers are incredible and somewhat hard to believe. It might be worthwhile to dig deep here and understand where such high numbers are coming from.
Again, thanks for all the work and I currently retain my recommendation.
Response to Reviewer WHpW
Thank you for reading our response and for raising this point. The end-to-end arm reaches 0.989–1.000 on MemAB-FC answers because the verdict of the check follows directly from the dependency map, and on MemAB-FC the Interpreter builds that map almost exactly.
Given a dependency map, the check of Definition 2.2 is a walk over the map without an LLM call, so its verdicts can be wrong only where the map is wrong. We compared the end-to-end map with the benchmark's structured updates fact by fact. At 6K, the two agree on the standing of all 455 facts. At 32K, they agree on 2,294 of the 2,310 facts (99.3%). The nine errors of the end-to-end arm all come from records that the Interpreter built wrongly. The same agreement explains Table 8, where the end-to-end arm is within 1.5 points of the native arm in QA accuracy. MemAB-FC states each fact in one short sentence, and a replacement restates the subject and relation with a new value, so the Interpreter's task is comparatively well defined.
The same evaluation gives lower numbers where the task is harder, so the evaluation itself is not lenient. When GPT-4o also performs the association, recall is 0.43–0.55 on multi-hop questions (Table 12), and on the free-form continuations we wrote, the check rejects 54 of the 57 ungrounded ones (Section 4.4).
Thank you again for your careful engagement with our work.
Response to Reviewer WHpW
Thank you for your careful review and for the clear questions! We ran the evaluation you asked for and answer your three questions first, then the two weaknesses.
Q1. Evaluating answers sampled from the QA tasks against independent labels.
Yes. We took the 3,280 distinct answers that the QA models gave in the submitted MemAB-FC experiments and labelled each one valid or unsupported (grounded and ungrounded in the paper). An answer is valid if a chain of records that still stand states its value, and unsupported otherwise. The labels come from the benchmark's own record of which facts were replaced, without running the verifier, and we reviewed a sample of 100 labels by hand against the records, stratified by label, and found all 100 correct.
We then ran the check of Definition 2.2 on every answer. Verifier (native) takes the records from the benchmark's structured updates, and Verifier (end-to-end) has the Interpreter build them from raw text. The end-to-end arm covers 6K and 32K, as in Table 8 of the revision.
The native arm takes its records from the same structured updates that the labels are derived from, so its score confirms that the implementation follows Definition 2.2. The end-to-end arm measures the method: the Interpreter builds the records from raw text, and every error of this arm comes from a record that it built wrongly. Section 4.4 of the revision reports this evaluation, and Table 4 gives the same counts as precision and recall.
Q2. How often unsupported continuations are rejected.
Given the benchmark's records, every unsupported answer is rejected. When the Interpreter builds the records from raw text, all 185 unsupported single-hop answers and 997 of the 1,003 unsupported multi-hop answers (0.994) are rejected. These are the last column of the table in Q1.
Q3. How often valid continuations are accepted.
Given the benchmark's records, every valid answer is accepted. When the Interpreter builds the records from raw text, 201 of the 202 valid single-hop answers (0.995) and 175 of the 177 valid multi-hop answers (0.989) are accepted. These are the third column of the table in Q1.
W1. Framing of the problem.
We agree. We removed the sentence on context-manipulation attacks from the abstract, so the abstract and the introduction now open with the same problem: continuations that rest on premises the conversation has abandoned. The introduction presents context manipulation as one source of such continuations.
W2. Direct evaluation when claim-to-record links and dependencies must be inferred.
Both cases are covered. The end-to-end arm in the table in Q1 is the case where the dependencies are inferred: 0.989–0.995 of the valid answers are accepted and 0.994–1.000 of the unsupported ones are rejected. For the link, which the paper calls association, both arms of that table use the entities and the relation named in the question. When GPT-4o infers the link instead, naming the record that an answer rests on without seeing which records still stand, no unsupported answer is accepted, and 0.961 of the valid single-hop answers (390 of 406) and 0.485 of the valid multi-hop answers (162 of 334) are accepted, pooled over the four stream lengths. Valid multi-hop answers are accepted less often because the LLM has to find the record at the end of a chain of facts.
The link is also inferred on free-form continuations. We wrote 86 of them on the two conversations of Section 3, the Phase 2 debugging session and the Phase 3 architecture deliberation (Appendix B.2 and B.3), 29 valid and 57 unsupported. With GPT-4o linking each continuation to a record, the check accepts all 29 valid continuations and rejects 54 of the 57 unsupported ones. When each continuation is instead linked to the record that shares the most words with it, the check accepts 26 of the 29 and rejects 48 of the 57, so the LLM link improves both. The protocol and the results under inferred links, per stream length and for two further LLMs, are in Appendix D.4 of the revision.
The list of revisions is in the general response. We are glad to run further checks during the discussion period.