- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
RAISE records facts that document summaries leave out and keeps a link from each fact to its source. An agent can search these facts and use them to decide which parts of a document to open. On eight document-QA benchmarks where the document is not given, RAISE achieves 43.1% average exact match compared with 37.3% for ViDoRAG.
Critical strengths
- Recording omitted facts gives the agent a way to find evidence that would otherwise be hidden by a summary, while the source links let it open the original content (Sections 2 and 4).
- The gains span several document collections, and removing the ledger lowers average exact match by 4.0 points, showing that it contributes to the result (Tables 1 and 3).
Critical weaknesses
- The no-ledger controls keep summaries written to rely on the ledger, so they do not show whether identifying omissions is better than indexing the same number of ordinary source facts (Appendices D.5 and E).
- The answer gate requires the agent to read a cited source but does not check that the source supports its answer, so an incorrect ledger fact may still mislead the agent (Section 4.2; Appendix E).
Questions for authors
- How does RAISE compare with indexing the same number of source facts without first checking what the summaries omitted?
- Have you checked how often an incorrect ledger fact leads to a wrong final answer even after the agent opens its source?
2 comments
Response to Reviewer PgBt (2/2)
[W2/Q2: How often an incorrect ledger fact leads to a wrong final answer after the agent opens its source]
W2. The answer gate requires the agent to read a cited source but does not check that the source supports its answer, so an incorrect ledger fact may still mislead the agent (Section 4.2; Appendix E).
Q2. Have you checked how often an incorrect ledger fact leads to a wrong final answer even after the agent opens its source?
To measure how often an incorrect ledger fact leads to a wrong final answer after the agent opens its source, we audited every ledger fact that the agent followed to its source on 500 queries per corpus, or all queries of a corpus that has fewer. The agent follows a fact when it reads the node the fact points to, after the fact reached the agent either as a result of the loss search or as a label on the map edge into that node. Claude Opus 5.5 compared each followed fact with the source text the agent read and marked it as supported by the source, contradicted by it, or absent from it. A fact counts as incorrect when it is contradicted or absent. We chose a judge stronger than the model that wrote the facts (Qwen3.6-35B-A3B), so that the facts are not judged by their own writer. The final answer is judged by the LLM judge of Appendix D.1.
| Ledger facts the agent followed to their source | n | final answer: correct | final answer: wrong | final answer: wrong-answer rate | value certified from the source |
|---|---|---|---|---|---|
| correct (supported by the source) | 8,435 | 5,510 | 2,925 | 34.7% | 44.6% |
| incorrect (contradicted or absent) | 1,616 | 1,025 | 591 | 36.6% | 40.7% |
| all followed facts | 10,051 | 6,535 | 3,516 | 35.0% | 44.0% |
| Queries, counted once each | n | share |
|---|---|---|
| exposed to at least one incorrect ledger fact | 841 | |
| answered wrongly | 279 | 33.2% |
| the answer repeats the error of the fact | 9 | 1.1% |
| the answer is wrong for a different reason | 268 | 31.9% |
| the records do not allow a judgment | 2 | 0.2% |
After the agent opens the source, a wrong final answer follows an incorrect ledger fact in 36.6% of the cases (591 of 1,616) against 34.7% after a correct fact (2,925 of 8,435). Several such facts can share the same final answer, so we also counted by query: 841 distinct queries were exposed to at least one incorrect fact, and 279 of them (33.2%) were answered wrongly. For each of the 279 we compared the fact, the source text the agent read, the value it certified, and the final answer. In 9 the answer repeats the error of the fact (1.1% of the exposed queries), in 268 the answer is wrong for a different reason, and in 2 the records do not allow a judgment. The agent certifies a value from the source in 40.7% of the cases after an incorrect fact and in 44.6% after a correct one.
The revised manuscript includes this audit as §5.5 and Table 4, with the query-level counts.
Response to Reviewer PgBt (1/2)
We thank the reviewer for the clear reading of the method and for two questions that each point at a measurable quantity. For W1/Q1, we compared the ledger with indexing the same number of ordinary source facts and re-indexed the corpora with summaries that do not rely on the ledger. For W2/Q2, an audit counts how often an incorrect ledger fact leads to a wrong final answer after the agent opens its source. The full tables are in the revised manuscript, which we have uploaded (Table 3, §5.4, §5.5, and Appendices F.2 and F.4).
[W1/Q1: Indexing the same number of ordinary source facts]
To test whether identifying omissions is better than indexing the same number of ordinary source facts, we added two controls on all eight corpora. (a) Indexing the same number of source facts without first checking what the summaries omitted, which keeps the summaries of the paper. (b) A re-indexing with a plain summarization prompt, so that the no-ledger control keeps summaries that were not written to rely on the ledger.
(a) In the control, we replaced the loss witnesses with facts extracted by one call per internal node with the same backbone, which sees the node's children and not the parent summary and receives no instruction about omissions. Two variants are kept. (i) The same number of facts per edge as the ledger holds there. (ii) The same cap per edge as the witness extraction, eight per child. The facts are embedded, placed in the loss channel and searched exactly as the witnesses are.
Identifying omissions is better than indexing the same number of ordinary source facts by 2.2 EM, and a 1.7 times larger index without the check recovers 0.5 EM more.
(i) With the same number of ordinary source facts per edge, accuracy falls to 40.9 EM, 2.2 points below the omitted facts and 1.8 above the no-ledger control (39.1). At the same per-edge count, ordinary source facts scored 40.9 EM and omission-aware witnesses scored 43.1. With the alternative summaries, adding the ledger improved EM from 39.4 to 42.7.
(ii) Under the cap of eight per child, the extractor tries to cover nearly everything in each child, so the index grows to 1.7 times the size of the ledger and accuracy recovers to 41.4 EM, still 1.7 points below the omitted facts.
(b) The summaries of the paper are written to rely on the ledger by sentences of the prompt of §F.2 that allow a summary to leave out what the ledger will hold. We removed such sentences and kept the 120-word budget and every other rule. The witnesses were re-extracted against the new summaries, and the no-ledger control and the full system were run again on all eight corpora.
The summaries written to rely on the ledger account for 0.7 EM of the ledger's 4.0 EM gain, and the ledger still gains 3.3 EM over a control whose summaries were not written for it: with such summaries the no-ledger control reaches 39.4 EM against 39.1 with the summaries of the paper, and the full system 42.7 against 43.1. Of the 0.7 EM, 0.3 is the handicap of the no-ledger control and 0.4 the full system's benefit from such summaries.
The revised manuscript adds the two rows of (a) and the two configurations of (b) to Table 3, discusses both in §5.4, and gives the two prompts in Appendices F.4 and F.2.