- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work studies a setting that combines non-factoid QA and long-document QA. For this setting, FactWeaver has two stages, i.e., Atomic Fact Graph Construction and Graph-Guided Multi-Agent Synthesis. The authors also build the LongNFQA dataset to provide a benchmark for this setting.
Critical strengths
- LongNFQA offers annotations from human reviewers for assessing the performance of both retrieval and answer quality. Besides, the LLM-based evaluator is validated against held-out human judgments.
- FactWeaver significantly outperforms all evaluated baseline systems. There are 11 ablation studies included in this paper for different components.
Critical weaknesses
- The baselines are not very convincing. Some baselines are different from their original versions; for example, RAPTOR has a simple hierarchy with TF-IDF retrieval, HippoRAG 2 does not use OpenIE, and Self-RAG and Search-R1 have prompted adaptations without their original training. There is no comparison with closely related approaches like MDocAgent and MA-RAG, nor a direct long-context LLM baseline. This weakens the argument that FactWeaver outperforms existing approaches more broadly.
- Gold evidence and the PS@20 metric are questionable. The gold evidence is chosen from candidates found during question construction and additional GPT-5.5 proposals, and then reviewed by humans. It is not clear how this process considers other possible evidence nodes from the documents, particularly since each question has only 7 to 11 gold nodes. Furthermore, PS@20 counts a perspective as satisfied only when all its required evidence nodes are retrieved.
- The graph relations obtained from the construction process are hardly analyzed. Sequence edges follow the order in local evidence plans, support edges use domain label matching, and conflict edges use semantic judgments. These rules may not always reflect meaningful relations between facts. While the ablations suggest that the edges are useful, there is little analysis provided about the actual relations used for reasoning.
- Some design decisions do not have enough justification. Optional agent routing uses keyword rules and fixed thresholds, while path selection relies on manually weighted scores. Ablations show the usefulness of the pipeline as a whole, but they do not offer much information on how these heuristics work. The cost of inference is quite high (676K tokens per question).
Questions for authors
- What impact do the baseline adaptations have on performance? Why are MDocAgent, MA-RAG, and long-context LLM baselines not considered?
- Do the annotators look for valid evidence beyond the model-proposed candidates?
- How sensitive is PS@20 to the requirement that all designated evidence nodes for a perspective be retrieved?
- Are conflict edges constructed within or across perspectives? The descriptions in Section 3.2 and Appendix C.2 appear inconsistent.
- How often are the optional agents invoked, and what percentage of issues are resolved during revision?
5 comments
Response to W3 & Q4: Relation meaning, selection and scope
We have grounded the three edge types in the discourse functions described by Mann and Thompson (1988), Rhetorical Structure Theory: Toward a Functional Theory of Text Organization. We have adapted these functions to relations between facts in an analytical answer. Conflict edges have been constructed within a perspective, while support edges have connected facts across perspectives. We have screened question relevance during graph construction, have assessed the connected facts and proposed statements during path selection, and have checked their use in the final answer.
The RST basis has guided the roles of our relations as follows:
Table H2. RST basis and analytical roles of the graph relations
| FactWeaver edge | RST basis | Role in the answer |
|---|---|---|
| Sequence | Sequence | Organize the order of facts in the explanation |
| Support | Evidence and Elaboration | Bring related facts together to support or explain a statement |
| Conflict | Contrast and Concession | Identify interpretations requiring reconciliation or qualification |
For graph construction, we have selected question-relevant facts and have used retrieved document evidence to relate them. We have formed sequence edges from the order within a perspective, support edges between perspectives addressing the same issue, and conflict edges from tensions within a perspective. This has tied the proposed connections to the question and its evidence.
For reasoning-path selection, the Planner has chosen facts and proposed statements using those relations. The Auditor has assessed whether the facts have supported the statements, whether required perspectives have been covered, and whether conflicts have been addressed. The Reviser has changed the fact selection, ordering or conclusions in response to the assessment before final path selection.
For the statements developed from the selected path, we have used the connected facts as evidence and have checked their joint interpretation during path assessment. The Writer has developed the answer from that path, and the Verifier has checked the answer against the selected evidence and accepted constraints. Together, these stages have provided multiple checks on the relevance and evidential support of the relations actually used in reasoning. The construction rules have supplied candidate connections, and their use in the answer has undergone further assessment.
Relations in the recorded reasoning. We have examined how the constructed relations have been retained in the selected paths across all 624 questions.
Table H3. Relations in graph construction and selected reasoning paths
| Relation | Candidate edges | Both facts selected | Both facts cited together |
|---|---|---|---|
| Sequence | 8,961 | 1,473 | 659 |
| Support | 2,457 | 629 | 246 |
| Conflict | 844 | 322 | 225 |
The columns have followed the constructed relations through selection of both facts and their joint use in a statement or explanation in the selected path. Final-answer verification has then assessed the written answer against that path. The counts have described relation use, while the construction and assessment steps have checked question relevance and evidential support.
The original wording has discussed cross-perspective connections before conflicts without making their different scopes clear. We have now specified cross-perspective support and within-perspective conflict separately in Section 3.2 and Appendix C.2, Table 10.
We have explained the RST rationale in those sections and the assessment and revision in Section 3.3 and Appendices C.3–C.4. Appendices F.1–F.2 have provided source-based analyses, and Appendix F.3 and Table 33 have reported the usage counts. Section 6.3 and Table 4 have provided the comparison without relational edges.
Response to W4: Routing thresholds, path selection and cost
We have used 0.75 to request additional review when more than one quarter of the selected perspectives have lacked facts, and two conflict edges to trigger further checking of multiple tensions. Evidence support has received a moderately larger weight than conflict resolution in path selection.
For additional review, we have used explicit comparison or causal questions to identify the need for further checking, together with the evidence represented in the graph. Every question has retained the fixed Auditor and Reviser. We have used the coverage and conflict thresholds to decide when more review has been needed.
Table H4. Reasons for the routing and selection settings
| Choice | Reason for the setting |
|---|---|
| Perspective coverage: 0.75 | Additional review after more than one quarter of the selected perspectives have lacked a fact |
| Extra Auditor: two conflict edges | An additional check for multiple recorded conflicts; a single conflict has already received the fixed audit and mediation |
| Evidence-support/conflict-resolution weights: 0.6/0.4 | A moderate preference for evidence support while retaining the need to resolve conflicting interpretations |
For the three-to-five-perspective setting, the coverage rule has required extra review after one missing perspective out of three, or two missing out of four or five. We have therefore used a proportion of missing analysis rather than the same missing-count rule for every question.
For final selection, we have first used the recommended path after checking it against the graph and audit findings. When it has failed those checks, we have compared the other paths. The evidence-support weighting has favored conclusions grounded in the selected facts, while the conflict-resolution weighting has retained the need for appropriate qualifications.
Table H5. Quality and token use associated with review steps
| Configuration | Mean Avg. | Online tokens (K) | Token saving versus full |
|---|---|---|---|
| FactWeaver | 4.4832 | 675.609 | 0.0% |
| w/o Conditional Roles | 4.3038 | 621.923 | 8.0% |
| w/o Path Assessment and Revision | 3.8118 | 574.918 | 14.9% |
Section 6.3 and Table 4 have reported the quality results, and Section 6.4 and Figure 2 have compared quality with token use. Appendix C.4 has explained the roles and path selection, and Appendix D.3 and Table 15 have listed the experimental settings.
Response to Q5: Optional-agent participation and revision outcomes
The Mediator, Extra Auditor and Domain Critic have participated on 93.91%, 36.86% and 87.66% of questions, respectively. Revision has corrected 66.64% of the initial missing perspectives and 87.50% of the initial references to facts outside the graph. It has also introduced new problems, which we have included in the comparison.
Table H6. Optional-agent participation across 624 questions
| Optional agent | Questions | Share |
|---|---|---|
| Mediator | 586 | 93.91% |
| Extra Auditor | 230 | 36.86% |
| Domain Critic | 547 | 87.66% |
We have combined the audit findings directly for the other 38 questions without the Mediator.
Across 1,125 original and revised reasoning-path pairs, we have measured:
Table H7. Changes across 1,125 original and revised path pairs
| Problem | Before revision | Corrected | Newly introduced | After revision | Correction rate for initial problems |
|---|---|---|---|---|---|
| Missing perspectives | 1,136 | 757 | 205 | 584 | 66.64% |
| References to facts outside the graph | 16 | 14 | 34 | 36 | 87.50% |
We have also found a missing perspective or a reference to a fact outside the graph in the final selected path for 154/624 questions (24.68%).
Appendix C.4, “Observed coordination and revision,” has reported these counts. We have used them to describe both the corrections achieved and the problems remaining after revision.
Manuscript revisions
We have made the following revisions in response to your comments:
- Sections 5.2 and 6.2, Appendices D.1 and D.4: Specified baseline adaptations and added MA-RAG and full-context comparisons.
- Sections 4.2 and 5.1, Appendix B.3: Clarified evidence review and the interpretation of PS@20 alongside recall.
- Sections 3.2–3.3, Appendices C.2–C.4 and F.1–F.3: Explained the RST motivation, relation assessment, within-perspective conflicts and observed relation use.
- Appendices C.4 and D.3, Section 6.4: Explained routing and selection settings, reported agent participation and revision outcomes, and presented the quality–token comparison.
Thank you for the detailed comments on the baselines, evidence annotations, graph relations and reasoning procedure.
Response to W1 & Q1: Task adaptations and added baselines
Our adaptations have aligned the baselines with the report-based task and shared generator, and the performance conclusion has remained supported by the comparisons with Typed-RAG, MA-RAG and full-context LLM. We have compared retrieval and reasoning within this task setting, and have added the requested MA-RAG and complete-document generation comparisons.
Task adaptations. We have used the same document evidence for local retrieval and the same generator for prompted reasoning. RAPTOR has tested retrieval through a document hierarchy, HippoRAG 2 has tested graph-based retrieval, and Self-RAG and Search-R1 have tested assessment and repeated search. Typed-RAG has provided the strongest baseline in the original comparison.
MDocAgent. We have prioritized analysis of the question and synthesis of evidence from text, tables and charts. MDocAgent has emphasized combining text and page images through specialized agents, which has differed from this comparison's focus.
MA-RAG. We have examined its original reasoning procedure and have completed the added comparison. The method has extracted information for individual factual subquestions and has combined their short answers. In our examples, useful evidence for the whole question has been discarded when it has not answered a narrower subquestion directly. Its original stopping rule has then ended some plans, and use of general knowledge has moved some answers beyond the report evidence. These observations have explained difficulties in applying that procedure to non-factoid synthesis. We have retained the original procedure and prompts in the added experiment.
Full-context LLM. We have tested direct answering from the complete report. Every question has retained the full parsed text, tables and chart content.
Table H1. Answer-quality comparison with the added baselines
We have described the adapted versions in Appendix D.1 and Table 12 and their common conditions in Appendix D.3. We have added the comparisons to Section 5.2 and Section 6.2, Table 3, and have described the new baselines in Appendix D.4.
Response to W2, Q2 & Q3: Gold evidence and PS@20
The 7–11 gold nodes per question have reflected the focused scope of each question within documents covering much broader material. Each node has represented a passage or whole table and has often contained several relevant facts. Annotators have checked related original text beyond the proposed excerpts when needed and have assessed the retained evidence against all parts of the question. PS@20 has required all designated evidence for a perspective, while recall has recorded partial recovery.
Evidence review. The question-construction evidence and GPT-5.5 proposals have supplied the initial candidates. Annotators have reviewed their content, have consulted the corresponding passages and related original context as needed, and have retained evidence supporting the requested analysis, including necessary qualifications. We have assessed sufficiency through coverage of the question's analytical requirements. The node count has described the passages and tables needed for that focused analysis, with each node potentially supporting several facts.
PS@20 sensitivity. Missing any required node has removed credit for its perspective. The loss for one previously satisfied perspective has equaled one divided by the number of annotated perspectives for the question. Additional omissions within an already unsatisfied perspective have produced no further decrease. Missing a relevant node outside that perspective's designated evidence has left its credit unchanged.
We have therefore used PS@20 to assess retrieval of all required evidence for each part of the answer and have reported precision, recall and F1 alongside it. The recall result has shown partial recovery of relevant nodes even when the stricter PS@20 condition has not been met.
We have clarified the review procedure and question-level sufficiency assessment in Section 4.2 and Appendix B.3. Section 5.1 and Equation 7 have defined PS@20, and Section 6.1 and Table 2 have reported it together with precision, recall and F1.