OG-ClaimWiki: Optimizing Knowledge Organization for LLM Agents through Evidence-Chain Probing

LLM agents often need to retrieve and connect multiple facts into sufficient evidence chains to answer complex questions. Knowledge organization affects whether agents can find these facts and establish the connections needed to support an answer. However, knowledge bases built using predefined rules or heuristic strategies may retain relevant facts without making these chains easy for agents to assemble. We introduce Offline-Grounded ClaimWiki (OG-ClaimWiki), a framework that uses evidence-chain probing to optimize knowledge organization before real user queries are available.OG-ClaimWiki represents source knowledge as verifiable atomic Claims and connects them through source-supported relations to form evidence chains. These chains guide probe generation and specify the evidence required for each answer. Comparing this evidence with the agent's access trace helps locate defects in knowledge content, search indexes, page links, and context organization. Candidate edits are verified against the sources and evaluated on the original probes, independent questions, and regression questions. Only validated edits are incorporated into the Wiki. Experiments on four document QA datasets and two code benchmarks demonstrate consistent improvements over the evaluated baselines. Offline probing increases document macro-average F1 by 2.6–4.3 points over the initial Wiki across answering models. OG-ClaimWiki also outperforms the strongest baselines in document and code QA accuracy after source changes.