acceptodds
Under review as a conference paper at ICLR 2027

OG-ClaimWiki: Optimizing Knowledge Organization for LLM Agents through Evidence-Chain Probing

Abstract

LLM agents often need to retrieve and connect multiple facts into sufficient evidence chains to answer complex questions. Knowledge organization affects whether agents can find these facts and establish the connections needed to support an answer. However, knowledge bases built using predefined rules or heuristic strategies may retain relevant facts without making these chains easy for agents to assemble. We introduce Offline-Grounded ClaimWiki (OG-ClaimWiki), a framework that uses evidence-chain probing to optimize knowledge organization before real user queries are available.OG-ClaimWiki represents source knowledge as verifiable atomic Claims and connects them through source-supported relations to form evidence chains. These chains guide probe generation and specify the evidence required for each answer. Comparing this evidence with the agent's access trace helps locate defects in knowledge content, search indexes, page links, and context organization. Candidate edits are verified against the sources and evaluated on the original probes, independent questions, and regression questions. Only validated edits are incorporated into the Wiki. Experiments on four document QA datasets and two code benchmarks demonstrate consistent improvements over the evaluated baselines. Offline probing increases document macro-average F1 by 2.6–4.3 points over the initial Wiki across answering models. OG-ClaimWiki also outperforms the strongest baselines in document and code QA accuracy after source changes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Knowledge Maintenance, Multi-Hop QA

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rpt5OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This article improves the performance of document Q&A and code Q&A by allowing the Agent to propose questions itself to find the locations of errors and then repair the knowledge base.

Critical strengths

This method is intuitive, from finding the error to fixing it, it conforms to the intuitive logic of traditional repair, and also improves performance.

Critical weaknesses

This method has not been clearly explained in terms of its practical significance in reality, as there are already many Harness architectures, such as Codex. In fact, by using a model with Codex, it can already answer questions from a knowledge base. They can even directly fix the knowledge base.

Questions for authors

On at least one Q&A task, directly use Codex and gpt-5.6 to search within the knowledge base and compare the performance of your method.

2 comments

Dear Reviewer rpt5,

We are writing to follow up on the rebuttal we submitted several days ago. Thank you again for your feedback. We believe our responses address the main concerns raised in your reviews. When you have a chance, we would be grateful for your thoughts on our responses, particularly whether any concerns remain. We would welcome further discussion and are happy to provide additional clarification. Thank you for your time and consideration.

Reasoning effort of GPT-5.6 Sol: What reasoning-effort setting was used for GPT-5.6 Sol in the Codex experiments (high or extra high)? Was the same setting used across all experimental conditions? If extra high was not used, could you also provide the results with extra-high reasoning effort?

Reviewer_JLeeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces OG-ClaimWiki, which generates questions from evidence chains and uses agent traces to diagnose and repair the Wiki. The system keeps edits only after source verification and QA validation. Experiments include document and code QA, with offline probing improving document Macro F1 by 2.6 to 4.3 points over the initial Wiki.

Critical strengths
  1. Diagnosis and repair have a clear basis. Comparing the required evidence with actual traces helps locate specific problems. Source verification and regression checks further reduce incorrect edits.
  2. The improvements are consistent. The same refined Wiki benefits multiple answering models. Improvements also extend to test questions whose supporting evidence is not covered by probe questions.
Critical weaknesses
  1. The sources of improvement are not sufficiently analyzed. It remains unclear how much of the gain comes from adding Claims, improving retrieval and page links, or introducing derived Claims. Statistics on edit types and changes in evidence acquisition would help explain the results.
  2. Some handling rules need clarification. Facts missing from the initial Claim graph may never lead to corresponding probe questions. The paper should also explain how the system accepts sufficient evidence that differs from the predefined chain.
Questions for authors
  1. What proportion of accepted edits belongs to each edit type? Does repair increase the proportion of questions for which the agent obtains the necessary evidence?
  2. Can the system discover facts or relations missing from the initial Claim graph? If not, how does this affect probe coverage?
  3. When an agent answers correctly through another sufficient evidence chain, does the system consider the attempt successful? How does it check the sufficiency of alternative evidence?
  4. Please provide the results for removing evidence-chain guidance and resolve the inconsistency between Section 4.4 and Figure 4.
Reviewer_pnSoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes OG-ClaimWiki, a framework for optimizing how a knowledge base is organized for LLM agents before real user queries are available. Sources (documents, tables, and code repositories) are decomposed into atomic, source-grounded Claims, connected by rule-based relations into a Claim graph, and compiled into Wiki pages with links and a search index. Paths in the Claim graph serve as evidence chains, from which the system generates single-Claim, relational, and compositional probe questions, keeping only those that none of the answering models can answer without retrieval. An agent answers each probe through the same search/read/navigate interface used at deployment, and its access trace is compared with the evidence chain to locate defects in Claim content, source bindings, the search index, page links, or context organization (repaired by adding derived Claims). Candidate local edits are committed only after passing source verification, answer recovery, independent validation, and regression checks. When sources change, updates are propagated to dependent Claims and links.

Experiments cover four document QA datasets (HotpotQA, MuSiQue, 2WikiMultiHopQA, HybridQA) and two code QA benchmarks (SWE-QA, SWE-QA-Pro), with five answering models sharing one refined Wiki. OG-ClaimWiki outperforms retrieval, graph-based, and Wiki/documentation baselines such as HippoRAG2, LLM-Wiki, CodeWiki, and RepoDoc. Relative to the no-probing Wiki, probing improves document Macro F1 by 2.6–4.3 points and code QA scores by 1.6–4.2 points. Claim-graph-based probing outperforms random and page-based probe generation at matched budgets, and directly injecting the probe QA pairs into the Wiki yields much smaller gains than diagnosis-guided repair. On source-update benchmarks (MQuAKE-T, CoDocBench), the method achieves the highest post-update accuracy (92.8% and 93.1%) while retaining about 99% of previously correct answers.

Critical strengths

1. A practical, model-agnostic problem setting.

The paper targets a realistic gap: improving a knowledge base before any user queries exist, without retraining the answering model. A single refined Wiki improves all five answering models on both document and code QA. Because each Claim records its source location and version, edits are verifiable and auditable, and dependency-aware updates are possible when sources change (Table 4).

2. Careful experimental controls.

Beyond the main comparison, the paper compares probe-generation strategies at matched question counts and Claim-count distributions (Table 3, Fig. 3). It separates diagnosis-guided repair from mere probe coverage with a probe-injection control (+3.1 vs. +0.9 Macro F1, App. D.5), analyzes gains by probe–test evidence overlap (App. D.4), tests diagnosis on injected defects (App. D.2), and reports bootstrap confidence intervals for key differences (App. D.7). These controls make the main claims more credible than end-to-end comparisons alone.

Critical weaknesses

1. The Claim graph is never evaluated directly.

The Claim graph drives the entire pipeline: probes are generated from its evidence chains (Sec. 3.2.2), diagnosis compares agent traces against these chains (Sec. 3.2.3), and validation questions are built from its Claims and relations (App. E.2). Yet the paper reports only end-to-end QA results. It provides neither basic statistics (e.g., the number of Claims and edges) nor any intrinsic quality measure, such as Claim-extraction precision/recall or the accuracy of entity alignment and relation edges; the human audit in App. B.2 covers the probes, not the graph. Facts missing from the graph can never be probed, and because the graph also serves as the reference for diagnosis and validation, the final QA numbers cannot show whether the gains reflect better knowledge organization in general or merely a Wiki that reproduces its own Claim graph more faithfully. Graph statistics and a human evaluation of sampled Claims and edges would help.

2. The self-generated probes may closely resemble the test questions.

Most probes are multi-hop chain questions that hide intermediate entities and have a unique answer (Sec. 3.2.2, App. B.2), and they are generated from the same corpus that contains the test evidence (App. D.4). HotpotQA, 2WikiMultiHopQA, and MuSiQue were themselves largely built by chaining facts through bridge entities. In effect, the system previews the exam format, prepares targeted answers (e.g., derived Claims that store multi-hop conclusions, Sec. 3.2.3), and then takes the exam. Since the test questions are never used (Sec. 4.1), this is not direct leakage, but it may amount to implicit adaptation to the benchmark distribution, an advantage that would not exist in a real deployment where future queries are unknown. App. D.4 checks only overlap in supporting paragraphs and near-duplicates, not similarity in question type or form. I suggest reporting accuracy separately for test questions that are similar vs. dissimilar to the probes, e.g., by question type (the test sets are already stratified by type, App. B.1) or by embedding similarity. The code results, whose test questions are open-ended, partly mitigate this concern.

3. The offline cost is high relative to the gains.

According to Table 18 (App. D.6), building and refining the document Wiki takes 1,697M tokens: 3.0× the no-probing Wiki, 1.9× LLM-Wiki, and 5.5× HippoRAG2. The code Wiki takes 640M tokens, 2.5–3.4× CodeWiki and RepoDoc. In return, probing adds 2.6–4.3 points of document Macro F1 and 1.6–4.2 points on code QA (Tables 1, 2, 11, 12). Candidate validation alone accounts for 52% of the document tokens, yet removing its independent-validation and regression checks lowers Macro F1 by only 0.3 (Fig. 4). Moreover, the cost analysis appears only in the appendix and omits monetary cost, wall-clock time, update cost, and corpus size, which makes it hard to judge scalability to realistic corpora. I remain somewhat skeptical that the gains justify the cost, and would like to see the cost discussed in the main text, along with comparisons in which baselines receive a matched offline budget.

Questions for authors

see Critical Weaknesses

4 comments

Dear Reviewer pnSo,

We are writing to follow up on the rebuttal we submitted several days ago. Thank you again for your thoughtful feedback. We believe our responses address the main concerns raised in your reviews. When you have a chance, we would be grateful for your thoughts on our responses, particularly whether any concerns remain. We would welcome further discussion and are happy to provide additional clarification. Thank you for your time and consideration.

Dear Reviewer pnSo,

As the discussion period draws to a close, we would be very grateful if you could review our rebuttal and let us know whether our responses have clarified the points raised in your review. Should any questions or concerns remain, we would welcome the opportunity to address them before the discussion ends.

Thank you again for your time and consideration.

Best regards, The Authors

Thank you for the detailed response. The new analyses, such as the results grouped by similarity to the probes, address part of my concerns. However, the following three issues remain.

  1. Small gains from the new components. The Wiki format and agent interface follow LLM-Wiki, and the compile–evaluate–refine loop follows WiCER. The new Claim representation alone improves over LLM-Wiki by only 1.0–1.5 Macro F1, and the two components that distinguish this work from prior methods each add only about one point: evidence-chain probing improves over page-based probing by 1.2 points (85.4 vs. 84.2), and removing access traces lowers Macro F1 by only 1.1. In addition, the ablation without evidence-chain guidance is still missing from Figure 4.

  2. Missing comparison with the most related work. The paper cites WiCER and DeepRefine but does not compare against them experimentally, so it is unclear whether the proposed probing and diagnosis improve over these methods. Several closely related works are also not discussed, including agent memory systems such as A-MEM (linking and memory evolution), Memory-R1 (editing memory based on QA outcomes), and Zep (invalidating outdated facts), as well as document expansion with doc2query, which corresponds to the probe-injection baseline.

  3. The cost remains high relative to the gains. The matched-budget comparison with LLM-Wiki is helpful, but given the limited changes noted in point 1, the method still requires about 1,697M tokens of frontier API calls, three times the cost of the no-probing Wiki. The code side is more telling: without probing, the initial Wiki does not consistently outperform CodeWiki or RepoDoc (e.g., 68.2 vs. 68.5 on SWE-QA with Claude Opus 5), so almost all of the gain comes from the expensive probing stage (640M vs. 220M tokens), which adds only 1.6–4.2 points.

For these reasons, I keep my current score now.

Thank you for the further feedback.

1. Gains from the new components

  • The gains are significant. The test sets contain 4,000 document questions and 980 code questions. On document QA, the Claim representation improves over LLM-Wiki by 1.5 points, about 60 more correctly answered questions, and probing adds 3.1 points, about 120 more. 95% confidence intervals from 10,000 paired bootstrap resamples show that the gain of the Claim representation over LLM-Wiki lies between 0.7 and 2.2 points, and the gain from probing lies between 2.5 and 3.8 points on documents and between 2.0 and 3.7 points on SWE-QA. All lower bounds are above zero, so the gains are not due to random variation.
  • Single-component ablations underestimate the new components. Page-QG also uses our trace-based diagnosis and validated repair, so 85.4 vs. 84.2 only measures building probes from evidence chains. Over the no-probing Wiki, Claim-QG gains 3.1 points, 1.6× the gain of Page-QG (+1.9). Claim-QG with half the budget outperforms Page-QG with the full budget.
  • Access traces mainly affect diagnosis accuracy. Removing them lowers Macro F1 by 1.1 points because most test questions are already answered correctly before repair. In the controlled-defect experiments, defect localization accuracy drops from 93.6% to 79.8%, and repair success drops from 88.7% to 76.3%.
  • Overall, OG outperforms LLM-Wiki by 4.6 points and removes 24% of the remaining errors.
  • We will add the evidence-chain ablation to Figure 4 in the revision.

2. Comparison with related work

  • WiCER has not released code, so we reimplemented its algorithm and added a comparison. WiCER selects one existing QA pair per source document as a fixed probe and evaluates on the same questions. Our setting has no existing QA pairs, and test questions are never used for repair. We therefore start from the same initial Wiki, sample as many source documents as OG has probes, and generate one fixed probe per document. For each failed probe, WiCER extracts the dropped facts and preserves them in recompilation, for at most two iterations, and we evaluate on the held-out test questions.
MethodDoc Macro F1Ret.SWE-QA
Initial Wiki82.3–68.5
WiCER83.497.969.2
OG85.499.471.4

WiCER gains only 1.1 points, and recompilation turns some previously correct answers wrong. WiCER can only restore dropped facts, while 48% of OG's edits modify search indexes and page links.

  • DeepRefine centers on training a refinement model with reinforcement learning, while OG trains no model. OG studies how to build probes from evidence chains, locate problems from access traces, and validate edits before any user queries exist. The two differ in goal and setting, so a direct comparison is difficult.
  • A-MEM, Memory-R1, and Zep are memory systems for conversational agents, while our benchmarks are given document collections and code repositories with no dialogue history.
    • A-MEM links each new memory to existing ones by semantic similarity and updates related old memories. These links are fixed at construction time, and A-MEM does not check whether an agent can follow them to the evidence. For such construction-time organization, we already compare with GraphRAG, LightRAG, HippoRAG 2, and LLM-Wiki.
    • Memory-R1 trains a memory manager with reinforcement learning, with rewards from QA outcomes against gold answers, which our setting does not have.
    • Zep uses a temporal knowledge graph to mark outdated facts as invalid. This corresponds to source updates, which we evaluate in a separate experiment.
  • doc2query corresponds to our probe-injection baseline. It gains only 0.9 points, compared with 3.1 for OG. On test questions not covered by probes, the gains are 0.3 and 1.7 points.

These works were not mentioned in the initial review, and our earlier response focused on the original comments. We will discuss them in the related work section of the revision.

3. Cost

  • For code QA, we correct our earlier statement. On SWE-QA, the no-probing code Wiki is comparable to the strongest baseline but not better for every model. On SWE-QA-Pro, it is the best for 4 of 5 models. The code QA gains come mainly from probing, which is the core contribution of this paper. Probing improves SWE-QA by 2.5–4.2 points for all five models, and the final results exceed the strongest baseline by 2.2–5.1 points.
  • Probing is a one-time offline cost. All five answering models share one Wiki. Half the probing budget achieves gains close to the full budget. Source updates cost only 7.1% (documents) and 15.6% (code) of a full build. After repair, Gemini 3.5 Flash outperforms Claude Opus 5 with the no-probing Wiki, so every later query can use a cheaper model.

Comments on the paper

1 comment

Dear AC and Reviewers,

Thank you for your reviews and discussion. As the discussion period is ending, we summarize our contributions and the discussion.

Contributions

  1. We optimize knowledge organization through evidence-chain probing before real user queries are available, so that agents can obtain sufficient evidence for answering.
  2. We diagnose and edit locally based on source evidence and agent access traces. Edits are retained only after source verification, independent-question validation, and regression checks.
  3. On document and code QA, one Wiki enables multiple answering models to outperform the baselines, and it is updated effectively after source changes.

Strengths recognized by the reviewers

Reviewers JLee and rpt5 recommend acceptance.

  • Diagnosis and repair have a clear basis: Comparing the required evidence with actual access traces locates specific problems, and source verification and regression checks reduce incorrect edits. The method follows an intuitive logic from finding errors to fixing them.
  • The improvements are consistent: The same Wiki benefits multiple answering models, and the gains extend to test questions whose evidence is not covered by probes.
  • The problem setting is practical and the experimental controls are careful: Reviewer pnSo also recognized that the problem setting is practical and model-agnostic, and that the matched-budget strategy comparisons, the probe-injection control, and the confidence intervals make the conclusions more credible.

Concerns of Reviewer pnSo and our responses

Reviewer pnSo noted that analyses such as the similarity grouping addressed part of the concerns. We responded to the remaining three points as follows:

  1. Gains from the new components. On 4,000 document questions, the gains from the Claim representation and from probing are both significant under bootstrap tests. Without access traces, defect localization accuracy drops from 93.6% to 79.8%.
  2. Comparison with related work. We reimplemented WiCER, which reaches 83.4 Macro F1, compared with 85.4 for OG. The other works require model training or dialogue histories and target different settings.
  3. Cost. When LLM-Wiki receives a larger budget, OG still uses 13% fewer tokens and scores 3.4 points higher.

We will include these experiments and discussions in the revision. Thank you again to all reviewers and the AC.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.