acceptodds
Under review as a conference paper at ICLR 2027

FactWeaver: Atomic Fact Graph-Guided Non-Factoid Question Answering over Long Documents

Abstract

Non-factoid questions seek explanations, comparisons, or assessments beyond a single fact. Answering them over long documents requires identifying dispersed facts relevant to a broad analytical question, determining their relationships, and integrating them into a coherent analysis. We introduce FactWeaver, an atomic fact graph-guided framework for non-factoid question answering over long documents. FactWeaver organizes question-relevant atomic facts under analytical perspectives and uses the retrieved context to establish candidate sequence, support, and conflict relations. The resulting atomic fact graph guides multiple agents in constructing candidate reasoning paths and auditing them for defects. Revision requirements are recorded in a shared state and guide answer generation. To evaluate the effectiveness of the framework, we construct LongNFQA, a benchmark comprising 624 non-factoid questions over 52 long documents totaling 6,048 pages. Human-reviewed evidence supports retrieval evaluation, while a human-derived multidimensional rubric and an automated judge validated against held-out human judgments support answer-quality assessment. On LongNFQA, FactWeaver improves Perspective Satisfaction@20 by 0.1384 and the Avg. answer-quality score by 0.5203 points over the strongest baseline. Ablation studies support the contributions of both atomic fact graph construction and graph-guided synthesis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Non-Factoid Question Answering, Long-Document Question Answering, Retrieval-Augmented Generation, Atomic Fact Graphs, Multi-Agent Reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_HXqkOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies a setting that combines non-factoid QA and long-document QA. For this setting, FactWeaver has two stages, i.e., Atomic Fact Graph Construction and Graph-Guided Multi-Agent Synthesis. The authors also build the LongNFQA dataset to provide a benchmark for this setting.

Critical strengths
  1. LongNFQA offers annotations from human reviewers for assessing the performance of both retrieval and answer quality. Besides, the LLM-based evaluator is validated against held-out human judgments.
  2. FactWeaver significantly outperforms all evaluated baseline systems. There are 11 ablation studies included in this paper for different components.
Critical weaknesses
  1. The baselines are not very convincing. Some baselines are different from their original versions; for example, RAPTOR has a simple hierarchy with TF-IDF retrieval, HippoRAG 2 does not use OpenIE, and Self-RAG and Search-R1 have prompted adaptations without their original training. There is no comparison with closely related approaches like MDocAgent and MA-RAG, nor a direct long-context LLM baseline. This weakens the argument that FactWeaver outperforms existing approaches more broadly.
  2. Gold evidence and the PS@20 metric are questionable. The gold evidence is chosen from candidates found during question construction and additional GPT-5.5 proposals, and then reviewed by humans. It is not clear how this process considers other possible evidence nodes from the documents, particularly since each question has only 7 to 11 gold nodes. Furthermore, PS@20 counts a perspective as satisfied only when all its required evidence nodes are retrieved.
  3. The graph relations obtained from the construction process are hardly analyzed. Sequence edges follow the order in local evidence plans, support edges use domain label matching, and conflict edges use semantic judgments. These rules may not always reflect meaningful relations between facts. While the ablations suggest that the edges are useful, there is little analysis provided about the actual relations used for reasoning.
  4. Some design decisions do not have enough justification. Optional agent routing uses keyword rules and fixed thresholds, while path selection relies on manually weighted scores. Ablations show the usefulness of the pipeline as a whole, but they do not offer much information on how these heuristics work. The cost of inference is quite high (676K tokens per question).
Questions for authors
  1. What impact do the baseline adaptations have on performance? Why are MDocAgent, MA-RAG, and long-context LLM baselines not considered?
  2. Do the annotators look for valid evidence beyond the model-proposed candidates?
  3. How sensitive is PS@20 to the requirement that all designated evidence nodes for a perspective be retrieved?
  4. Are conflict edges constructed within or across perspectives? The descriptions in Section 3.2 and Appendix C.2 appear inconsistent.
  5. How often are the optional agents invoked, and what percentage of issues are resolved during revision?
5 comments

Thank you for the detailed comments on the baselines, evidence annotations, graph relations and reasoning procedure.


Response to W1 & Q1: Task adaptations and added baselines

Our adaptations have aligned the baselines with the report-based task and shared generator, and the performance conclusion has remained supported by the comparisons with Typed-RAG, MA-RAG and full-context LLM. We have compared retrieval and reasoning within this task setting, and have added the requested MA-RAG and complete-document generation comparisons.

Task adaptations. We have used the same document evidence for local retrieval and the same generator for prompted reasoning. RAPTOR has tested retrieval through a document hierarchy, HippoRAG 2 has tested graph-based retrieval, and Self-RAG and Search-R1 have tested assessment and repeated search. Typed-RAG has provided the strongest baseline in the original comparison.

MDocAgent. We have prioritized analysis of the question and synthesis of evidence from text, tables and charts. MDocAgent has emphasized combining text and page images through specialized agents, which has differed from this comparison's focus.

MA-RAG. We have examined its original reasoning procedure and have completed the added comparison. The method has extracted information for individual factual subquestions and has combined their short answers. In our examples, useful evidence for the whole question has been discarded when it has not answered a narrower subquestion directly. Its original stopping rule has then ended some plans, and use of general knowledge has moved some answers beyond the report evidence. These observations have explained difficulties in applying that procedure to non-factoid synthesis. We have retained the original procedure and prompts in the added experiment.

Full-context LLM. We have tested direct answering from the complete report. Every question has retained the full parsed text, tables and chart content.

Table H1. Answer-quality comparison with the added baselines

MethodQwen Avg.DeepSeek Avg.Two-model mean
Typed-RAG4.06973.49093.7803
Native MA-RAG2.47122.36782.4195
Full-context LLM3.77563.47223.6239
FactWeaver4.59004.37634.4832

We have described the adapted versions in Appendix D.1 and Table 12 and their common conditions in Appendix D.3. We have added the comparisons to Section 5.2 and Section 6.2, Table 3, and have described the new baselines in Appendix D.4.


Response to W2, Q2 & Q3: Gold evidence and PS@20

The 7–11 gold nodes per question have reflected the focused scope of each question within documents covering much broader material. Each node has represented a passage or whole table and has often contained several relevant facts. Annotators have checked related original text beyond the proposed excerpts when needed and have assessed the retained evidence against all parts of the question. PS@20 has required all designated evidence for a perspective, while recall has recorded partial recovery.

Evidence review. The question-construction evidence and GPT-5.5 proposals have supplied the initial candidates. Annotators have reviewed their content, have consulted the corresponding passages and related original context as needed, and have retained evidence supporting the requested analysis, including necessary qualifications. We have assessed sufficiency through coverage of the question's analytical requirements. The node count has described the passages and tables needed for that focused analysis, with each node potentially supporting several facts.

PS@20 sensitivity. Missing any required node has removed credit for its perspective. The loss for one previously satisfied perspective has equaled one divided by the number of annotated perspectives for the question. Additional omissions within an already unsatisfied perspective have produced no further decrease. Missing a relevant node outside that perspective's designated evidence has left its credit unchanged.

We have therefore used PS@20 to assess retrieval of all required evidence for each part of the answer and have reported precision, recall and F1 alongside it. The recall result has shown partial recovery of relevant nodes even when the stricter PS@20 condition has not been met.

We have clarified the review procedure and question-level sufficiency assessment in Section 4.2 and Appendix B.3. Section 5.1 and Equation 7 have defined PS@20, and Section 6.1 and Table 2 have reported it together with precision, recall and F1.

Response to W3 & Q4: Relation meaning, selection and scope

We have grounded the three edge types in the discourse functions described by Mann and Thompson (1988), Rhetorical Structure Theory: Toward a Functional Theory of Text Organization. We have adapted these functions to relations between facts in an analytical answer. Conflict edges have been constructed within a perspective, while support edges have connected facts across perspectives. We have screened question relevance during graph construction, have assessed the connected facts and proposed statements during path selection, and have checked their use in the final answer.

The RST basis has guided the roles of our relations as follows:

Table H2. RST basis and analytical roles of the graph relations

FactWeaver edgeRST basisRole in the answer
SequenceSequenceOrganize the order of facts in the explanation
SupportEvidence and ElaborationBring related facts together to support or explain a statement
ConflictContrast and ConcessionIdentify interpretations requiring reconciliation or qualification

For graph construction, we have selected question-relevant facts and have used retrieved document evidence to relate them. We have formed sequence edges from the order within a perspective, support edges between perspectives addressing the same issue, and conflict edges from tensions within a perspective. This has tied the proposed connections to the question and its evidence.

For reasoning-path selection, the Planner has chosen facts and proposed statements using those relations. The Auditor has assessed whether the facts have supported the statements, whether required perspectives have been covered, and whether conflicts have been addressed. The Reviser has changed the fact selection, ordering or conclusions in response to the assessment before final path selection.

For the statements developed from the selected path, we have used the connected facts as evidence and have checked their joint interpretation during path assessment. The Writer has developed the answer from that path, and the Verifier has checked the answer against the selected evidence and accepted constraints. Together, these stages have provided multiple checks on the relevance and evidential support of the relations actually used in reasoning. The construction rules have supplied candidate connections, and their use in the answer has undergone further assessment.

Relations in the recorded reasoning. We have examined how the constructed relations have been retained in the selected paths across all 624 questions.

Table H3. Relations in graph construction and selected reasoning paths

RelationCandidate edgesBoth facts selectedBoth facts cited together
Sequence8,9611,473659
Support2,457629246
Conflict844322225

The columns have followed the constructed relations through selection of both facts and their joint use in a statement or explanation in the selected path. Final-answer verification has then assessed the written answer against that path. The counts have described relation use, while the construction and assessment steps have checked question relevance and evidential support.

The original wording has discussed cross-perspective connections before conflicts without making their different scopes clear. We have now specified cross-perspective support and within-perspective conflict separately in Section 3.2 and Appendix C.2, Table 10.

We have explained the RST rationale in those sections and the assessment and revision in Section 3.3 and Appendices C.3–C.4. Appendices F.1–F.2 have provided source-based analyses, and Appendix F.3 and Table 33 have reported the usage counts. Section 6.3 and Table 4 have provided the comparison without relational edges.

Response to W4: Routing thresholds, path selection and cost

We have used 0.75 to request additional review when more than one quarter of the selected perspectives have lacked facts, and two conflict edges to trigger further checking of multiple tensions. Evidence support has received a moderately larger weight than conflict resolution in path selection.

For additional review, we have used explicit comparison or causal questions to identify the need for further checking, together with the evidence represented in the graph. Every question has retained the fixed Auditor and Reviser. We have used the coverage and conflict thresholds to decide when more review has been needed.

Table H4. Reasons for the routing and selection settings

ChoiceReason for the setting
Perspective coverage: 0.75Additional review after more than one quarter of the selected perspectives have lacked a fact
Extra Auditor: two conflict edgesAn additional check for multiple recorded conflicts; a single conflict has already received the fixed audit and mediation
Evidence-support/conflict-resolution weights: 0.6/0.4A moderate preference for evidence support while retaining the need to resolve conflicting interpretations

For the three-to-five-perspective setting, the coverage rule has required extra review after one missing perspective out of three, or two missing out of four or five. We have therefore used a proportion of missing analysis rather than the same missing-count rule for every question.

For final selection, we have first used the recommended path after checking it against the graph and audit findings. When it has failed those checks, we have compared the other paths. The evidence-support weighting has favored conclusions grounded in the selected facts, while the conflict-resolution weighting has retained the need for appropriate qualifications.

Table H5. Quality and token use associated with review steps

ConfigurationMean Avg.Online tokens (K)Token saving versus full
FactWeaver4.4832675.6090.0%
w/o Conditional Roles4.3038621.9238.0%
w/o Path Assessment and Revision3.8118574.91814.9%

Section 6.3 and Table 4 have reported the quality results, and Section 6.4 and Figure 2 have compared quality with token use. Appendix C.4 has explained the roles and path selection, and Appendix D.3 and Table 15 have listed the experimental settings.


Response to Q5: Optional-agent participation and revision outcomes

The Mediator, Extra Auditor and Domain Critic have participated on 93.91%, 36.86% and 87.66% of questions, respectively. Revision has corrected 66.64% of the initial missing perspectives and 87.50% of the initial references to facts outside the graph. It has also introduced new problems, which we have included in the comparison.

Table H6. Optional-agent participation across 624 questions

Optional agentQuestionsShare
Mediator58693.91%
Extra Auditor23036.86%
Domain Critic54787.66%

We have combined the audit findings directly for the other 38 questions without the Mediator.

Across 1,125 original and revised reasoning-path pairs, we have measured:

Table H7. Changes across 1,125 original and revised path pairs

ProblemBefore revisionCorrectedNewly introducedAfter revisionCorrection rate for initial problems
Missing perspectives1,13675720558466.64%
References to facts outside the graph1614343687.50%

We have also found a missing perspective or a reference to a fact outside the graph in the final selected path for 154/624 questions (24.68%).

Appendix C.4, “Observed coordination and revision,” has reported these counts. We have used them to describe both the corrections achieved and the problems remaining after revision.


Manuscript revisions

We have made the following revisions in response to your comments:

  • Sections 5.2 and 6.2, Appendices D.1 and D.4: Specified baseline adaptations and added MA-RAG and full-context comparisons.
  • Sections 4.2 and 5.1, Appendix B.3: Clarified evidence review and the interpretation of PS@20 alongside recall.
  • Sections 3.2–3.3, Appendices C.2–C.4 and F.1–F.3: Explained the RST motivation, relation assessment, within-perspective conflicts and observed relation use.
  • Appendices C.4 and D.3, Section 6.4: Explained routing and selection settings, reported agent participation and revision outcomes, and presented the quality–token comparison.
Reviewer_uUPoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies non-factoid question answering over long documents, where relevant evidence is dispersed across narrative text, tables, and charts and must be identified, related, and integrated into a coherent analysis. The authors propose FactWeaver: it first builds an atomic fact graph in which analytical perspectives organize atomic facts and candidate relations are induced, then multiple agents construct, audit, and revise reasoning paths under a shared state before answer generation. The authors also construct LongNFQA, a benchmark that fills the gap of combining bounded long documents, non-factoid questions, and human-reviewed source-level evidence for retrieval evaluation. On LongNFQA, FactWeaver consistently outperforms the best-performing baseline among those evaluated in the paper in both retrieval and answer quality.

Critical strengths
  • The research problem is real and valuable. LongNFQA provides sufficiently fine-grained source-level evidence annotations and multidimensional quantitative evaluation metrics; this evaluation suite has the potential to become a standard benchmark for the field.
  • Under the paper's own evaluation protocol, the full framework consistently outperforms other methods across multiple answer-quality dimensions, with pronounced gains on more verifiable factuality-related metrics such as FNC, consistent with the improvement trend in the deterministic retrieval metrics. The graph structure also provides reasoning-level traceability that facilitates hallucination checking.
  • Fixing retrieved evidence while removing relational edges (w/o RE, −0.5601 Avg., −0.7179 RS) isolates the graph's contribution rather than conflating it with better retrieval, which strengthens the central claim.
Critical weaknesses
  • FactWeaver consumes about 675.6K online tokens per question, higher than most baselines (e.g., Naive RAG: 68.7K; Typed-RAG: 589.7K). While a few baselines use even more tokens, they are not the strongest competitors; compared with Typed-RAG, the strongest baseline, FactWeaver still uses more tokens. Moreover, in ablation experiments where token consumption is close to Typed-RAG (e.g., w/o CS, w/o PC, w/o PAR), the metrics do not show clear superiority and are even slightly lower. This raises concerns about inference cost, latency, and deployability.
  • Generalization is unverified: all conclusions rest on a single self-constructed benchmark, and without cross-framework comparisons on other datasets, it is impossible to tell whether the advantages extend beyond the distribution produced by its own construction pipeline.
  • The answer-quality judge is Qwen3.5-Flash—the same model family as the answer generator (Section C, A.3). Although the authors argue the held-out human comparison provides an "independent reference," that validation was performed over the original 15-system set; the six supplemental ablations were evaluated by the automatic judge only, after the protocol was frozen, so the largest quality drops (e.g., w/o AFA −1.2652) are never human-checked. Self-preference bias cannot be excluded for these comparisons.
  • All conclusions are conditional on Qwen3.5-Flash and all-mpnet-base-v2. There is no evidence about whether the FactWeaver-vs-baseline ordering survives under a different generator family, which matters for a framework whose selling point is agent orchestration. Even a small-scale replication (e.g., one document family, one alternative LLM) would substantially strengthen generalizability claims.
Questions for authors
  • The ablations show that the three variants closest to Typed-RAG's token budget (w/o CS, w/o PC, w/o PAR) do not clearly outperform it. This makes me wonder whether the key driver of the gains is the audit-and-revision itself or the graph structure: could you evaluate a control in which Typed-RAG is augmented with an iterative self-review/revision at a comparable token budget (simply reviewing the answer multiple times at the generation stage)? This bears on the assessed value of introducing the graph structure to this problem.
  • Could you provide a transfer evaluation on at least one external dataset (e.g., M-LongDoc)? This would substantially address my second weakness.
  • Table 5 shows lower judge–human agreement for Focus & Clarity (ρ = 0.715) than for the other five dimensions (ρ ≥ 0.90), which the paper attributes to its greater subjectivity. Could you provide concrete examples showing (1) how this disagreement manifests in practice, and (2) whether it materially affects the reliability of evaluation on the Focus & Clarity dimension?
6 comments

Thank you for your careful assessment of the quality–cost tradeoff, generalization and evaluation reliability.


Response to W1 & Q1: Quality, inference cost and iterative Typed-RAG

We have completed the requested iterative Typed-RAG control on all 624 LongNFQA questions. Answer review and revision have improved Typed-RAG, while FactWeaver has retained higher mean answer quality, particularly in Evidence Grounding and Reasoning Soundness. We have included the additional inference in the cost comparison: FactWeaver has achieved its quality advantage with higher online token use.

For iterative Typed-RAG, we have retained the original answer and retrieved evidence and have added up to three review–revision rounds. Review has checked missing analysis, evidence support, factual and numerical errors, reasoning and citations. We have stopped when review has found no further problems and have limited the additional inference to 100,000 tokens per question. This comparison has directly tested additional answer refinement using the same retrieved evidence.

Table U1. Answer quality and online token use on LongNFQA

MethodMean Avg.Mean Evidence GroundingMean Reasoning SoundnessOnline tokens (K)
Typed-RAG3.78033.74043.6442589.736
Iterative Typed-RAG4.11653.99284.0737631.510
FactWeaver4.48324.59544.5112675.609
FactWeaver − Iterative Typed-RAG+0.3667+0.6026+0.4375+44.099

We have averaged the quality scores from Qwen3.5-Flash and DeepSeek-V4.1-Flash. We have calculated iterative Typed-RAG's mean token use by combining its original mean with the added review and revision, and have included every question in the comparison.

The control in Table U1 has shown a benefit from answer review, while the remaining differences in evidence grounding and reasoning soundness have supported the value of the full method. We have added the answer-quality comparison in Section 6.2 and Table 3, the quality–token comparison in Section 6.4 and Figure 2, and the control description in Appendix D.4 and Table 16. Appendix E.3, Tables 22–24, has provided further cost results, and Appendix E.5 and Table 30 have reported paired confidence intervals.

Response to W2, W4 & Q2: External dataset and alternative generator

We have tested transfer beyond LongNFQA and Qwen generation on 48 original non-factoid questions from 14 selected M-LongDoc documents. We have used DeepSeek-V4.1-Flash as the generator for all five methods and have adopted M-LongDoc's Correctness criterion. FactWeaver has achieved the highest overall score under both evaluation models (Table U3).

The original LongNFQA evaluation has addressed the shortage of benchmarks combining long documents, non-factoid questions and suitable evidence annotations. For the external experiment, we have selected documents and questions from M-LongDoc's validated test set, retaining the original question wording and document assignments.

We have used the NF-CATS classifier from A Non-Factoid Question-Answering Taxonomy (SIGIR 2022) to assist selection, then have checked the reasoning required by each retained question. We have selected questions requiring explanation, comparison, conditional reasoning or multi-step instruction. We have excluded questions whose main task has been a single-value or entity lookup or a simple calculation.

We have selected multiple eligible questions from the same document, with two to four questions per document, and have balanced the three document domains:

Table U2. Selected M-LongDoc documents and questions

DomainDocument typeDocumentsQuestions
AcademicResearch papers416
FinancialCompany reports516
ProductProduct manuals516
Total—1448

We have fixed the subset before running the methods. All five methods have had access to the complete parsed documents, including text, tables and chart information. DeepSeek has performed FactWeaver's document analysis, question processing, evidence organization and generation. We have retained all-mpnet-base-v2 retrieval and the MiniLM reranker. The official evidence-page annotations have been reserved for evaluation.

For scoring, we have adopted the Correctness criterion from M-LongDoc, Section 2.3 and Appendix A.4. Qwen and DeepSeek have each rated how accurately and completely the answer has addressed the question using the annotated evidence-page content and adjacent context. We have used document evidence without a reference answer and have retained the original 1–5 scale.

Table U3. Correctness on the selected M-LongDoc questions

MethodQwen CorrectnessDeepSeek CorrectnessTwo-model mean
Naive RAG2.89584.04173.4688
Full-context LLM3.29173.97923.6354
Typed-RAG3.08334.10423.5938
Iterative Typed-RAG3.39584.31253.8542
FactWeaver3.72924.72924.2292

We have added the experiment to Section 6.5 and Table 5. Appendix G.1 has described selection, document access and alternative generation; Appendix G.2 has explained the adopted scoring criterion; and Appendix E.6 and Table 31 have provided results by evaluator and domain. This experiment has provided evidence with both external questions and a different generator.

Response to W3: Additional evaluator and human-validation coverage

We have evaluated all systems with Qwen and the additional DeepSeek model, including the six later ablations. Under both evaluation models, all eleven ablation comparisons have retained the same direction.

For the original 15 systems, the reported Qwen results have used adjudicated human scores on 168 questions and model scores on the remaining 456. The six later ablations and three added baselines have used model scores on all 624 questions. DeepSeek has evaluated all 24 systems.

Both evaluation models have assessed the same answers and human-reviewed evidence with the same six-dimensional rubric:

Table U4. Selected ablation results under both evaluation models

SystemQwen Avg.DeepSeek Avg.
FactWeaver4.59004.3763
w/o Atomic-Fact Acquisition3.32482.8232
w/o Graph Representation3.73992.9586
w/o Collaborative Synthesis3.44982.8408
w/o Relational Edges4.02993.7417
w/o Path Assessment and Revision3.95993.6637

We have reported the complete scores in Appendix E.2, Tables 20–21, and results for each evaluation criterion in Appendix E.4 and Figure 4. Section 6.3 and Table 4 have reported the mean of the two evaluators. Appendix A.1 has distinguished the human and model score sources.


Response to Q3: Focus & Clarity disagreements and their effect

FC disagreements have involved scores 4 and 5 over repetition, organization and readability. The DOE examples have shown both a closing summary that has repeated earlier points and a dense answer presented in one paragraph. Removing FC has improved judge–human ranking agreement, while FactWeaver's mean advantage over Typed-RAG has remained.

Both examples have answered the same question: “What evidence in DOE's FY 2023 AFR connects mission performance, financial management status, and IIJA resource execution?”

In the CoRAG answer, the opening has introduced “integrated reporting frameworks, audit validation, and strategic dashboarding.” The four-paragraph body has explained mission alignment, audit assurance, spending obligations and the Lifecycle Dashboard. The closing has returned to “rigorous audit assurance, detailed obligation tracking, and specialized digital dashboards.” This overlap has illustrated the distinction between a useful recap and avoidable repetition. The human reviewers have also differed between the two adjacent scores.

In the w/o Conditional Roles answer, the explanation has combined audit assurance, dashboards, grants, loan commitments and liabilities in one paragraph and has used “active enabler” twice. We have used this example to illustrate a continuous explanation that has also been dense to read.

Table U5. FC scores for two answers to the same DOE question

Answer to DOE FY 2023 question 01Reviewer AReviewer BAdjudicatedJudge
CoRAG5445
w/o Conditional Roles5554

These cases have illustrated both directions of disagreement using actual answers. Across all 1,260 held-out answers, the 304 FC disagreements have all involved scores 4 and 5.

We have then examined the effect on ranking agreement and method means:

Table U6. Sensitivity to removing Focus & Clarity

CheckWith FCWithout FC
Judge–human ranking correlation, 84 held-out questions0.88950.9195
FactWeaver Qwen-based method mean, 624 questions4.5900≈4.551
Typed-RAG Qwen-based method mean, 624 questions4.0697≈4.014

We have computed the approximate five-criterion means from the reported Qwen criterion means. The results have preserved FactWeaver's mean advantage while showing FC's contribution to lower judge–human agreement. We have retained the original six-criterion evaluation and have included these examples in Appendix A.5 and Table 7 and the sensitivity analysis in Appendix A.6.


Manuscript revisions

We have made the following revisions in response to your comments:

  • Sections 6.2 and 6.4, Appendix D.4: Added iterative Typed-RAG and compared answer quality with online token use.
  • Section 6.5, Appendices E.6 and G: Added evaluation on original M-LongDoc questions with DeepSeek generation and both evaluators.
  • Section 5.1, Appendices A.1 and E.2: Added DeepSeek evaluation and clarified the scope of human and model scoring.
  • Appendices A.5–A.6: Added concrete FC disagreement examples and analysis of their effect on ranking agreement and method means.
Reviewer_49TaOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

FactWeaver is a framework for answering complex, non-factoid questions over long documents. It identifies question-relevant analytical perspectives, extracts atomic facts, retrieves supporting evidence, and organizes the facts into a graph. Multiple agents propose and revise reasoning paths before generating an answer.

Critical strengths

The paper addresses an important and challenging problem: answering complex questions that require combining evidence from different parts of long documents.

The evaluation is well structured, assessing both evidence retrieval and answer quality through multiple baselines and extensive component ablations. Human-reviewed evidence annotations and held-out validation of the automated judge strengthen the results

Critical weaknesses

Comment 1: The main paper does not provide sufficient detail to understand key parts of the methodology, making the appendix necessary rather than supplementary.

To name a few examples:

Section 3.2 describes perspectives abstractly, while the concrete example listing three selected perspectives appears only in Appendix F.1

Likewise, the main text states that facts across perspectives receive support edges when they share “question-relevant context,” without explaining how this is determined. The operational rule—assigning question-grounded domain labels and linking facts with matching non-generic labels—appears only in Appendix C.2.

Moving these methodological details and a worked example into the main paper would substantially improve clarity and allow readers to understand how the graph is constructed.

Comment 2: The origins of the 52 reports are not clearly specified in the paper beyond describing them as publicly accessible professional reports. Including this information in the paper would make the dataset easier to understand and its source diversity easier to assess.

Comment 3: The evaluation uses Qwen3.5-Flash for both answer generation and automated scoring. Testing models from different families would help establish whether the reported gains generalize beyond this model and reduce concerns about potential bias from using the same model to generate and evaluate answers.

Questions for authors
  1. How are perspectives defined and selected in practice?
  2. How does the system determine that facts from different perspectives jointly support a claim?
  3. Which organizations and sources supplied the 52 reports, and what criteria guided their selection?
3 comments

Thank you for your careful reading and helpful suggestions on the method explanation, document sources and evaluation.


Response to Comment 1, Q1 & Q2: Method details and worked example

We have expanded Section 3.2 with the definition and selection of perspectives and the roles of sequence, support and conflict edges. We have also brought a worked example from Appendix F.1 into the main text, connecting the selected perspectives and retrieved facts with the audit findings and revised answer.

Perspective definition and selection. We have defined a perspective as a document-supported part of the analysis required to answer a question. An LLM has identified candidate perspectives in the document's sections and has associated each with supporting passages. We have combined perspectives with the same analytical role. For each question, the LLM has selected three to five perspectives according to their relevance to the requested analysis and coverage of its different parts, while avoiding unnecessary repetition.

Joint support. An LLM has assigned each fact to the question-related issue it has addressed, and we have linked facts from different perspectives assigned to the same issue. The Planner has proposed a statement from the linked facts, the Auditor has checked whether those facts have supported it, and the Reviser has corrected or qualified unsupported statements. The assessment has examined the content of the facts and the conclusion drawn from them.

In the Codex example, one fact has stated the general use of JECFA toxicological advice in maximum-level decisions, and another has described a JECFA arsenic assessment. The selected path has connected the general principle with the specific scientific assessment. In the DOE funding example, the Auditor has rejected an unsupported statement treating the audit opinion as a prerequisite for funding, and the final answer has distinguished funding mechanisms from audit assurance.

Worked example. The DOE example in the main text has covered financial reporting and internal control, accounting standards and liabilities, and operational risk and payment integrity. It has included a query about the improper-payment rate and has shown how assessment and revision have restored missing evidence and corrected an unsupported liability statement.

We have added the explicit definition and worked example in Section 3.2, with the selection principle in Equation 3. Appendix C.2 and Table 10 have explained relation construction, Section 3.3 and Appendix C.3 have described assessment and revision, and Appendices F.1–F.2 and Table 32 have provided the detailed examples.


Response to Comment 2 & Q3: Report sources and selection

We have downloaded the 52 reports from official company websites, U.S. government and agency websites, international organizations' websites, and official technical-publication pages. The anonymous repository has already provided source information and download links for every report. For ethical and copyright reasons, we have provided the original documents through these links, since some publishers have restricted redistribution of their original PDFs.

The sources have included company report pages, SEC company filings, government-agency reports, and publications from the World Bank, WHO and IAEA. The anonymous repository has identified the reporting organizations, publication types, editions and official links, allowing readers to identify and obtain the documents.

For selection, we have required English long-form reports with narrative text and structured evidence, analytical content distributed across sections, and usable correspondence between the extracted content and the original document. We have selected reports supporting questions that require combining evidence, and have varied organizations, activities, technical subjects and editions within each family.

Table T1. Document families in LongNFQA

Document familyPDFsQuestionsAnalytical reading conditions
Financial/business22264Statements, business explanations, resources and risks
Scientific/technical896Measurements, technical assessments and qualifications
Medical/public-health896Program funding, payment integrity and oversight
Policy/public-sector896Objectives, expenditure, implementation and accountability
Regulatory/safety672Requirements, thresholds, applicability and exceptions

We have expanded the selection criteria in Section 4.1 and Appendix B.2, and Table 9 has explained the contribution of each document family. We have added the “Document access and redistribution” paragraph in Appendix B.2 to explain the existing source links and the publishers' redistribution conditions.

Response to Comment 3: Additional evaluator and alternative generator

We have added DeepSeek-V4.1-Flash as an additional evaluation model, and Qwen3.5-Flash and DeepSeek-V4.1-Flash have each evaluated the same LongNFQA answers. We have also added an external M-LongDoc experiment with DeepSeek generation. FactWeaver has achieved higher mean scores in both comparisons (Table T2).

Additional evaluation model. Both models have assessed all 24 systems on the same LongNFQA questions using the six-dimensional rubric and human-reviewed evidence.

Alternative generator. We have run five methods on unchanged M-LongDoc questions with DeepSeek-V4.1-Flash. Both evaluators have assessed the resulting answers using M-LongDoc Correctness.

Table T2. Results with two evaluation models and alternative generation

Setting and metricMethodQwenDeepSeekTwo-model mean
LongNFQA: Avg.Typed-RAG4.06973.49093.7803
LongNFQA: Avg.FactWeaver4.59004.37634.4832
M-LongDoc: CorrectnessTyped-RAG3.08334.10423.5938
M-LongDoc: CorrectnessFactWeaver3.72924.72924.2292

We have added the evaluation model in Section 5.1 and the results in Section 6.2 and Table 3. Appendix E.2, Tables 20–21, has provided the complete LongNFQA results. Section 6.5 and Table 5 have presented the external experiment; Appendix G has described its conditions, and Appendix E.6 and Table 31 have provided results by evaluator and domain.


Manuscript revisions

We have made the following revisions in response to your comments:

  • Section 3.2: Added the perspective definition, clarified selection and graph relations, and included a worked example.
  • Section 4.1 and Appendix B.2: Expanded the report-selection criteria and explained access to the original documents through source links.
  • Section 5.1, Section 6.2 and Appendix E.2: Added DeepSeek evaluation and reported the results from both evaluation models.
  • Section 6.5 and Appendix G: Added the M-LongDoc experiment with an alternative generator.

Thank you again for your thoughtful review. As the discussion period is nearing its end, we would like to kindly follow up on our rebuttal and ensure that we have adequately addressed your concerns.

We would greatly appreciate any further feedback you may have, and we would be happy to clarify any remaining questions.

Thank you for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.