- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Sensemaker, a capability-modular multi-agent framework for multi-document analytical question answering (MDocAQA). The method adopts a three-layer architecture consisting of a task layer, a capability layer, and a tool layer, assigning file localization, file profiling, content extraction, data processing, and evidence verification to specialized agents, with the task layer dynamically coordinating them. Experiments cover six benchmarks and three backbone models, and Sensemaker achieves the highest average score in all settings. Module-level and architecture-level ablations further support the effectiveness of the design.
Critical strengths
-
The paper addresses agent overload and the difficulty of inspecting execution in multi-document analysis by designing capability modules with clearly defined responsibilities and structured interfaces. Each module executes independently and uses dedicated tools, giving the overall architecture clear structure and practical value.
-
Experiments cover six benchmarks and three backbone models, and Sensemaker achieves the highest average score in all 18 evaluation settings. Table 4 further provides module-level and architecture-level ablations, including controls such as the general subagent and single-session role switching, making the experimental evaluation relatively thorough.
Critical weaknesses
-
The necessity of the capability partitioning and process traceability still lack direct validation. Although Table 4 supports the effectiveness of independent capability modules, it does not examine the effects of different ways of partitioning or merging modules. Moreover, the paper lists process opacity as a main research problem but does not directly evaluate error localization or process inspection.
-
The statistical information and evidence verification of the experimental results also remain insufficient. Section 4.1 states that each experimental configuration is run three times, but Table 2 and Table 4 report only means, lacking standard deviations or confidence intervals. The file-level metrics in Appendix D.2 rely on source paths reported by the agents and still lack independent verification based on actual execution records.
Questions for authors
-
Can different ways of partitioning capability modules be compared, and can the improvement in process traceability brought by modularization be directly evaluated using error-localization metrics?
-
Can standard deviations or confidence intervals be provided for Table 2 and Table 4 to confirm the stability of the performance gains?
-
Can the file provenance in the final answers be verified against tool-call records, and can its consistency with the agent-reported results be analyzed?
4 comments
Response to Reviewer x7K7 [Part 2/2]
Standard Deviations of Experimental Results (Weakness 2(1) & Question 2)
Thank you for highlighting the importance of standard deviations in evaluating the stability and reliability of repeated experiments. We now include the standard deviations over three runs alongside the mean results in Tables 2 and 4. Given the character limit for each response, only partial results are shown here; the full results are available in Tables 2 and 4 of the revised paper. Across the comparisons presented here, Sensemaker achieves the highest score and the lowest standard deviations among the agent-based baseline methods and evaluated variants. These findings support the effectiveness of explicit capability modeling in enabling stable and accurate performance on MDocAQA tasks.
Table 2. Experimental results of agent-based methods with GLM-5.2.
| Method | Loong | xAFS | FRAMES | MuDA |
|---|---|---|---|---|
| LongAgent | 72.2${\pm1.21}$ | 38.2 | 31.8 | 53.9{\\pm1.50}\ |
| CoA | 71.2${\pm1.42}$ | 36.7 | 28.6 | 51.8{\\pm1.72}\ |
| MAIN-RAG | 61.3${\pm1.64}$ | 43.9 | 24.3 | 59.0{\\pm1.86}\ |
| BELLE | 73.2${\pm0.92}$ | 49.7 | 36.3 | 65.1{\\pm1.37}\ |
| LongTA | 74.7${\pm0.69}$ | 53.9 | 44.2 | 62.0{\\pm1.13}\ |
| Sensemaker (ours) | 79.9${\pm0.27}$ | 67.0 | 64.6 | 72.3{\\pm0.65}\ |
Table 4. Performance investigation of Sensemaker.
| Variant | |
|---|---|
| Sensemaker | 66.7${\pm0.44}$ |
| w/o dedicated FL | 54.8${\pm0.81}$ |
| w/o dedicated CE | 52.2${\pm0.73}$ |
| w/o dedicated DP | 49.2${\pm0.79}$ |
| w/o capability instantiation | 45.5${\pm0.94}$ |
| w/ general subagent | 41.8${\pm1.05}$ |
| w/o capability layer | 40.9${\pm1.52}$ |
Verification of File Recall and Precision (Weakness 2(2) & Question 3)
We appreciate your concern about the reliability of the R and P calculation. Our approach is reliable because agents must cite supporting files for each conclusion, so compiling the contributing files only requires collecting references already present in the answer, with little additional reasoning. To further verify this reliability, we compared our approach with a direct, non-LLM-based calculation. Tool-call records may include files explored but not cited in the final answer, so directly counting these files is not accurate. Therefore, we used regular expressions to extract the file paths cited in the final answers and recomputed R and P. Due to the response character limit, the table below shows only partial results. For the 192 paired R and P values across all settings, the two calculation strategies yield a concordance correlation coefficient (CCC) of 0.997 and a mean absolute error (MAE) of 0.955 percentage points. This close agreement supports the reliability of our evaluation method.
Table. Comparison of two strategies under GLM-5.2. Values before and after “/” correspond to the strategy used in the paper and the non-LLM-based strategy, respectively.
| Method | xAFS R | xAFS P | FRAMES R | FRAMES P | MuDA R | MuDA P |
|---|---|---|---|---|---|---|
| LongAgent | 72.6/72.9 | 58.1/58.6 | 32.5/33.8 | 33.8/33.6 | 69.2/67.4 | 55.7/55.1 |
| CoA | 69.3/67.8 | 71.8/71.6 | 30.2/30.7 | 35.6/33.7 | 64.8/64.7 | 58.9/60.2 |
| MAIN-RAG | 64.2/65.4 | 85.2/85.8 | 33.8/33.8 | 49.6/48.1 | 49.7/49.0 | 81.2/81.4 |
| BELLE | 71.6/73.3 | 78.1/79.0 | 42.2/42.9 | 45.5/46.0 | 62.9/64.7 | 74.2/75.2 |
| LongTA | 76.2/76.3 | 80.3/81.0 | 44.5/43.2 | 47.5/48.5 | 84.3/84.0 | 83.4/82.5 |
| Sensemaker (ours) | 89.0/88.9 | 87.5/86.8 | 81.8/83.5 | 72.9/74.1 | 93.7/95.1 | 89.4/89.6 |
Dear Reviewer x7K7,
Thank you again for your thoughtful feedback. We would like to follow up on our rebuttal and the revised manuscript. Please let us know if any concerns remain unresolved or if further clarification would be helpful. We would also appreciate it if you could revisit your assessment and rating in light of the clarifications and additional evidence we have provided.
Thank you for the time and effort you have devoted to reviewing our work.
Best regards,
All authors
Dear Reviewer x7K7,
As the discussion period draws to a close, we would be grateful if you could take a moment to read our response and let us know whether it adequately addresses your concerns. If so, we hope you will consider updating your rating accordingly.
Thank you again for your time and consideration.
Best regards,
All authors
Response to Reviewer x7K7 [Part 1/2]
We sincerely thank you for your positive and detailed review. Below, we address your concerns point-by-point.
Design Motivation of Sensemaker (Weakness 1(1) & Question 1(1))
We appreciate the suggestion to compare Sensemaker with variants of module merging. Drawing on the model of expert sensemaking discussed in Appendix A, Sensemaker organizes multi-document analysis into five capabilities. FL locates relevant files, FP characterizes specific files, and CE extracts required content, supporting information acquisition and evidence construction in the foraging loop. DP processes collected data and EV checks whether claims are supported by cited sources, supporting the sensemaking loop. In particular, FL and FP serve different purposes: FL discovers candidate files across the collection, while FP provides reusable descriptions of file contents and roles to guide subsequent analysis.
We randomly sampled 100 failed trajectories from the baseline methods (20 samples from each of the five agent-based methods.) and manually counted occurrences of six error types: missing relevant files (E1), misjudging file attributes or roles (E2), content extraction errors (E3), data processing errors (E4), unsupported or misattributed claims (E5), and other errors (E6). These error types are not mutually exclusive: a single trajectory may contain multiple types of errors. As shown in the table below, the predominant errors in these trajectories concern the responsibilities explicitly modeled by the five capabilities. This provides empirical motivation for their selection.
Table. Failure analysis of existing methods.
To further examine the capability boundaries, we evaluated four additional variants: merging FL and FP; merging CE and DP; merging FL, FP, and CE; and distributing evidence verification across the other modules. As shown in the table below, all four achieve lower answer scores than Sensemaker. Together with the five module-removal variants reported in the paper, we can find that FL primarily improves file coverage, FP improves relevance discrimination, CE supports effective and efficient evidence extraction, DP improves downstream data processing, and EV improves answer quality at a modest token cost. These results support both explicit capability modeling and the chosen boundaries.
Table. Performance investigation.
We have added the analysis in Appendix A in the revised paper.
Evaluation of Process Opacity (Weakness 1(2) & Question 1(2))
We sincerely thank the reviewer for the valuable suggestion regarding evaluation of process opacity. As we discussed in the Introduction section, process opacity arises when operations with different purposes and their intermediate results are mixed within a single trace, making results harder to inspect and errors harder to locate. Sensemaker separates these operations into capability modules with clear responsibilities, each running in an independent session. This makes intermediate results easier to inspect and helps trace errors to the relevant modules, narrowing the scope of diagnosis and correction.
Using GLM-5.2 as the backbone model, we randomly selected 100 questions answered correctly by all methods and manually injected errors into the corresponding execution traces. We then asked GLM-5.2 to locate these errors and measured the diagnosis success rate and average diagnosis time. As shown in Table 8, Sensemaker achieves the highest diagnosis success rate and the shortest average diagnosis time. These results suggest that its execution traces are easier to inspect and errors easier to locate, providing empirical evidence that Sensemaker alleviates process opacity.
Table 8. Results of trajectory diagnosis.
We have added these analyses to Appendix I.2 in the revised paper.