1. The Claim graph is never evaluated directly.
The Claim graph drives the entire pipeline: probes are generated from its evidence chains (Sec. 3.2.2), diagnosis compares agent traces against these chains (Sec. 3.2.3), and validation questions are built from its Claims and relations (App. E.2). Yet the paper reports only end-to-end QA results. It provides neither basic statistics (e.g., the number of Claims and edges) nor any intrinsic quality measure, such as Claim-extraction precision/recall or the accuracy of entity alignment and relation edges; the human audit in App. B.2 covers the probes, not the graph. Facts missing from the graph can never be probed, and because the graph also serves as the reference for diagnosis and validation, the final QA numbers cannot show whether the gains reflect better knowledge organization in general or merely a Wiki that reproduces its own Claim graph more faithfully. Graph statistics and a human evaluation of sampled Claims and edges would help.
2. The self-generated probes may closely resemble the test questions.
Most probes are multi-hop chain questions that hide intermediate entities and have a unique answer (Sec. 3.2.2, App. B.2), and they are generated from the same corpus that contains the test evidence (App. D.4). HotpotQA, 2WikiMultiHopQA, and MuSiQue were themselves largely built by chaining facts through bridge entities. In effect, the system previews the exam format, prepares targeted answers (e.g., derived Claims that store multi-hop conclusions, Sec. 3.2.3), and then takes the exam. Since the test questions are never used (Sec. 4.1), this is not direct leakage, but it may amount to implicit adaptation to the benchmark distribution, an advantage that would not exist in a real deployment where future queries are unknown. App. D.4 checks only overlap in supporting paragraphs and near-duplicates, not similarity in question type or form. I suggest reporting accuracy separately for test questions that are similar vs. dissimilar to the probes, e.g., by question type (the test sets are already stratified by type, App. B.1) or by embedding similarity. The code results, whose test questions are open-ended, partly mitigate this concern.
3. The offline cost is high relative to the gains.
According to Table 18 (App. D.6), building and refining the document Wiki takes 1,697M tokens: 3.0× the no-probing Wiki, 1.9× LLM-Wiki, and 5.5× HippoRAG2. The code Wiki takes 640M tokens, 2.5–3.4× CodeWiki and RepoDoc. In return, probing adds 2.6–4.3 points of document Macro F1 and 1.6–4.2 points on code QA (Tables 1, 2, 11, 12). Candidate validation alone accounts for 52% of the document tokens, yet removing its independent-validation and regression checks lowers Macro F1 by only 0.3 (Fig. 4). Moreover, the cost analysis appears only in the appendix and omits monetary cost, wall-clock time, update cost, and corpus size, which makes it hard to judge scalability to realistic corpora. I remain somewhat skeptical that the gains justify the cost, and would like to see the cost discussed in the main text, along with comparisons in which baselines receive a matched offline budget.
Dear Reviewer rpt5,
We are writing to follow up on the rebuttal we submitted several days ago. Thank you again for your feedback. We believe our responses address the main concerns raised in your reviews. When you have a chance, we would be grateful for your thoughts on our responses, particularly whether any concerns remain. We would welcome further discussion and are happy to provide additional clarification. Thank you for your time and consideration.