acceptodds
Under review as a conference paper at ICLR 2027

Don't Wear Too Many Hats: Capability Modularization for Multi-Document Analytical Question Answering

Abstract

Multi-document analytical question answering (MDocAQA) requires coordination among different information processing capabilities over evidence distributed across files. In existing methods, individual agent typically needs to perform operations across multiple capabilities. Since single agent often struggles to sustain effective reasoning and reflection throughout long-horizon workflows that require coordinating diverse capabilities, its performance tends to be poor. To address this issue, we propose an MDocAQA framework called Sensemaker. Inspired by the model of expert sensemaking, which represents information analysis as an iterative process containing distinct information processing activities, Sensemaker organizes MDocAQA into three layers (i.e., task layer, capability layer, and tool layer) and explicitly defines five modules responsible for different information processing capabilities in the capability layer. During task execution, the task-layer agent dynamically coordinates these modules, each instantiated as a specialized agent equipped with dedicated tools. This explicit capability modularization reduces the workload on individual agent and makes its intermediate states easier to inspect. Extensive experimental results on various benchmarks show that Sensemaker outperforms representative MDocAQA methods, with up to a 29.71% relative improvement in average answer score over the strongest baseline method.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi-Agent Framework, Multi-Document Analytical Question Answering, Capability Modularization

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_x7K7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Sensemaker, a capability-modular multi-agent framework for multi-document analytical question answering (MDocAQA). The method adopts a three-layer architecture consisting of a task layer, a capability layer, and a tool layer, assigning file localization, file profiling, content extraction, data processing, and evidence verification to specialized agents, with the task layer dynamically coordinating them. Experiments cover six benchmarks and three backbone models, and Sensemaker achieves the highest average score in all settings. Module-level and architecture-level ablations further support the effectiveness of the design.

Critical strengths
  1. The paper addresses agent overload and the difficulty of inspecting execution in multi-document analysis by designing capability modules with clearly defined responsibilities and structured interfaces. Each module executes independently and uses dedicated tools, giving the overall architecture clear structure and practical value.

  2. Experiments cover six benchmarks and three backbone models, and Sensemaker achieves the highest average score in all 18 evaluation settings. Table 4 further provides module-level and architecture-level ablations, including controls such as the general subagent and single-session role switching, making the experimental evaluation relatively thorough.

Critical weaknesses
  1. The necessity of the capability partitioning and process traceability still lack direct validation. Although Table 4 supports the effectiveness of independent capability modules, it does not examine the effects of different ways of partitioning or merging modules. Moreover, the paper lists process opacity as a main research problem but does not directly evaluate error localization or process inspection.

  2. The statistical information and evidence verification of the experimental results also remain insufficient. Section 4.1 states that each experimental configuration is run three times, but Table 2 and Table 4 report only means, lacking standard deviations or confidence intervals. The file-level metrics in Appendix D.2 rely on source paths reported by the agents and still lack independent verification based on actual execution records.

Questions for authors
  1. Can different ways of partitioning capability modules be compared, and can the improvement in process traceability brought by modularization be directly evaluated using error-localization metrics?

  2. Can standard deviations or confidence intervals be provided for Table 2 and Table 4 to confirm the stability of the performance gains?

  3. Can the file provenance in the final answers be verified against tool-call records, and can its consistency with the agent-reported results be analyzed?

4 comments
AuthorsOpenReview ↗

Response to Reviewer x7K7 [Part 1/2]

We sincerely thank you for your positive and detailed review. Below, we address your concerns point-by-point.

Design Motivation of Sensemaker (Weakness 1(1) & Question 1(1))

We appreciate the suggestion to compare Sensemaker with variants of module merging. Drawing on the model of expert sensemaking discussed in Appendix A, Sensemaker organizes multi-document analysis into five capabilities. FL locates relevant files, FP characterizes specific files, and CE extracts required content, supporting information acquisition and evidence construction in the foraging loop. DP processes collected data and EV checks whether claims are supported by cited sources, supporting the sensemaking loop. In particular, FL and FP serve different purposes: FL discovers candidate files across the collection, while FP provides reusable descriptions of file contents and roles to guide subsequent analysis.

We randomly sampled 100 failed trajectories from the baseline methods (20 samples from each of the five agent-based methods.) and manually counted occurrences of six error types: missing relevant files (E1), misjudging file attributes or roles (E2), content extraction errors (E3), data processing errors (E4), unsupported or misattributed claims (E5), and other errors (E6). These error types are not mutually exclusive: a single trajectory may contain multiple types of errors. As shown in the table below, the predominant errors in these trajectories concern the responsibilities explicitly modeled by the five capabilities. This provides empirical motivation for their selection.

Table. Failure analysis of existing methods.

Error TypeCountCorresponding Sensemaker capabilities
E163FL: Identify and retain candidates that warrant further exploration
E247FP: Accurately describe the file and its purpose
E339CE: Obtain accurate and traceable facts
E458DP: Properly process and integrate existing data
E531CE (information source binding) and EV (information source verification)
E68Other errors (tool execution, budget caps, task understanding, and so on)

To further examine the capability boundaries, we evaluated four additional variants: merging FL and FP; merging CE and DP; merging FL, FP, and CE; and distributing evidence verification across the other modules. As shown in the table below, all four achieve lower answer scores than Sensemaker. Together with the five module-removal variants reported in the paper, we can find that FL primarily improves file coverage, FP improves relevance discrimination, CE supports effective and efficient evidence extraction, DP improves downstream data processing, and EV improves answer quality at a modest token cost. These results support both explicit capability modeling and the chosen boundaries.

Table. Performance investigation.

VariantSco.Sco.Sco.(gold)Sco._{(gold)}RRPPT.Num.T.Num.
Sensemaker66.7±0.4477.5±0.3583.6±0.5777.0±0.510.405±0.008
merging FL and FP56.6±0.8775.2±0.6665.9±0.8362.8±0.710.484±0.014
merging CE and DP50.9±0.6969.7±0.8076.1±0.7268.4±0.600.692±0.018
merging FL, FP, and CE52.2±0.6167.4±0.9663.2±1.2456.9±1.110.737±0.020
moving EV into the others59.4±1.1573.1±0.8480.3±0.9572.2±1.260.951±0.023

We have added the analysis in Appendix A in the revised paper.

Evaluation of Process Opacity (Weakness 1(2) & Question 1(2))

We sincerely thank the reviewer for the valuable suggestion regarding evaluation of process opacity. As we discussed in the Introduction section, process opacity arises when operations with different purposes and their intermediate results are mixed within a single trace, making results harder to inspect and errors harder to locate. Sensemaker separates these operations into capability modules with clear responsibilities, each running in an independent session. This makes intermediate results easier to inspect and helps trace errors to the relevant modules, narrowing the scope of diagnosis and correction.

Using GLM-5.2 as the backbone model, we randomly selected 100 questions answered correctly by all methods and manually injected errors into the corresponding execution traces. We then asked GLM-5.2 to locate these errors and measured the diagnosis success rate and average diagnosis time. As shown in Table 8, Sensemaker achieves the highest diagnosis success rate and the shortest average diagnosis time. These results suggest that its execution traces are easier to inspect and errors easier to locate, providing empirical evidence that Sensemaker alleviates process opacity.

Table 8. Results of trajectory diagnosis.

MethodSuccess RateAverage Diagnosis Time (second)
LongAgent0.69957.02
CoA0.641024.31
MAIN-RAG0.72642.83
BELLE0.75612.94
LongTA0.79843.85
Sensemaker (ours)0.94371.26

We have added these analyses to Appendix I.2 in the revised paper.

AuthorsOpenReview ↗

Response to Reviewer x7K7 [Part 2/2]

Standard Deviations of Experimental Results (Weakness 2(1) & Question 2)

Thank you for highlighting the importance of standard deviations in evaluating the stability and reliability of repeated experiments. We now include the standard deviations over three runs alongside the mean results in Tables 2 and 4. Given the character limit for each response, only partial results are shown here; the full results are available in Tables 2 and 4 of the revised paper. Across the comparisons presented here, Sensemaker achieves the highest score and the lowest standard deviations among the agent-based baseline methods and evaluated variants. These findings support the effectiveness of explicit capability modeling in enabling stable and accurate performance on MDocAQA tasks.

Table 2. Experimental results of agent-based methods with GLM-5.2.

MethodLoongxAFSFRAMESMuDA
LongAgent72.2${\pm1.21}$38.2pm1.49{\\pm1.49}31.8pm1.62{\\pm1.62}53.9{\\pm1.50}\
CoA71.2${\pm1.42}$36.7pm1.55{\\pm1.55}28.6pm1.77{\\pm1.77}51.8{\\pm1.72}\
MAIN-RAG61.3${\pm1.64}$43.9pm1.71{\\pm1.71}24.3pm2.09{\\pm2.09}59.0{\\pm1.86}\
BELLE73.2${\pm0.92}$49.7pm1.13{\\pm1.13}36.3pm1.48{\\pm1.48}65.1{\\pm1.37}\
LongTA74.7${\pm0.69}$53.9pm1.13{\\pm1.13}44.2pm1.44{\\pm1.44}62.0{\\pm1.13}\
Sensemaker (ours)79.9${\pm0.27}$67.0pm0.43{\\pm0.43}64.6pm0.70{\\pm0.70}72.3{\\pm0.65}\

Table 4. Performance investigation of Sensemaker.

VariantSco.Sco.
Sensemaker66.7${\pm0.44}$
w/o dedicated FL54.8${\pm0.81}$
w/o dedicated CE52.2${\pm0.73}$
w/o dedicated DP49.2${\pm0.79}$
w/o capability instantiation45.5${\pm0.94}$
w/ general subagent41.8${\pm1.05}$
w/o capability layer40.9${\pm1.52}$

Verification of File Recall and Precision (Weakness 2(2) & Question 3)

We appreciate your concern about the reliability of the R and P calculation. Our approach is reliable because agents must cite supporting files for each conclusion, so compiling the contributing files only requires collecting references already present in the answer, with little additional reasoning. To further verify this reliability, we compared our approach with a direct, non-LLM-based calculation. Tool-call records may include files explored but not cited in the final answer, so directly counting these files is not accurate. Therefore, we used regular expressions to extract the file paths cited in the final answers and recomputed R and P. Due to the response character limit, the table below shows only partial results. For the 192 paired R and P values across all settings, the two calculation strategies yield a concordance correlation coefficient (CCC) of 0.997 and a mean absolute error (MAE) of 0.955 percentage points. This close agreement supports the reliability of our evaluation method.

Table. Comparison of two strategies under GLM-5.2. Values before and after “/” correspond to the strategy used in the paper and the non-LLM-based strategy, respectively.

MethodxAFS RxAFS PFRAMES RFRAMES PMuDA RMuDA P
LongAgent72.6/72.958.1/58.632.5/33.833.8/33.669.2/67.455.7/55.1
CoA69.3/67.871.8/71.630.2/30.735.6/33.764.8/64.758.9/60.2
MAIN-RAG64.2/65.485.2/85.833.8/33.849.6/48.149.7/49.081.2/81.4
BELLE71.6/73.378.1/79.042.2/42.945.5/46.062.9/64.774.2/75.2
LongTA76.2/76.380.3/81.044.5/43.247.5/48.584.3/84.083.4/82.5
Sensemaker (ours)89.0/88.987.5/86.881.8/83.572.9/74.193.7/95.189.4/89.6

Dear Reviewer x7K7,

Thank you again for your thoughtful feedback. We would like to follow up on our rebuttal and the revised manuscript. Please let us know if any concerns remain unresolved or if further clarification would be helpful. We would also appreciate it if you could revisit your assessment and rating in light of the clarifications and additional evidence we have provided.

Thank you for the time and effort you have devoted to reviewing our work.

Best regards,

All authors

Dear Reviewer x7K7,

As the discussion period draws to a close, we would be grateful if you could take a moment to read our response and let us know whether it adequately addresses your concerns. If so, we hope you will consider updating your rating accordingly.

Thank you again for your time and consideration.

Best regards,

All authors

Reviewer_UgisOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the multi-document analytical question answering (MDocAQA) task, a relatively novel problem scenario. This task requires agents to locate, extract, and process information from large collections of documents, perform cross-document reasoning and computation, and ultimately generate accurate answers. The main limitations of existing methods are agent overload and opaque processes. To address these challenges, the authors propose a new multi-agent framework inspired by the model of expert sensemaking. By organizing the system into three layers (i.e., task layer, capability layer, and tool layer) and explicitly defines five modules responsible for different information processing capabilities in the capability layer, the proposed framework enables individual agents to focus on their specialized responsibilities and perform tasks efficiently. The framework is easy to follow and well-structured. Extensive experiments on various MDocAQA benchmarks and representative baseline methods demonstrate its effectiveness.

Critical strengths
  1. The paper identifies two important limitations of existing MDocAQA methods, namely agent overload and process opacity, and provides a simple yet effective solution to address them. I greatly appreciate the authors' clear writing, which allowed me to quickly grasp the core arguments and methodological overview.
  2. Sensemaker adopts a clear architecture with three layers and five specialized capability modules. The responsibilities of individual modules are explicitly defined, making the overall framework easy to understand and potentially extensible.
  3. The experimental part is compelling. The authors conduct comprehensive experiments on six benchmarks using three state-of-the-art LLMs and compare Sensemaker against seven representative baselines, demonstrating consistent performance improvements across different settings.
Critical weaknesses
  1. The proposed framework introduces five specialized capability modules, but the reasons behind this design is not sufficiently explained. For instance, why are file localization and file profiling treated as separate capabilities, and how were these five capabilities determined to be effective for MDocAQA? Although the experiments demonstrate its performance, I am still very curious about the motivation behind this design.
  2. The paper identifies agent overload and process opacity as two major limitations of existing MDocAQA methods. While the proposed framework achieves performance improvements, it remains unclear to what extent these improvements can be attributed to addressing the two identified issues.
  3. The authors state that each experiment is repeated three times, but only the mean results are reported. Given that some improvements over the baselines are relatively small, the absence of standard deviations makes it difficult to assess the stability and reliability of these improvements.
Questions for authors
  1. The authors mention that each experiment is repeated three times (Line 317), but only the mean results are reported. Could the authors also provide the standard deviations to demonstrate the stability of the reported improvements, especially in settings where the performance differences are relatively small? I believe that if the authors indeed conducted three independent runs, calculating the standard deviations should be straightforward and would significantly strengthen the paper's credibility. Since the authors emphasize that their method outperforms all baselines across numerous experiments (Line 369), I strongly recommend that they report the standard deviations or more detailed results.
  2. While MuDABench (MuDA) (Li et al., 2026b) also adopts a multi-agent workflow with specialized components, Sensemaker introduces a more flexible architecture. Could the authors further clarify the key methodological differences and explain the unique benefits over existing multi-agent workflows?
  3. Could the authors provide more detailed discussion for the proposed framework? Specifically, what are the fundamental causes of agent overload and process opacity in existing MDocAQA methods, and why can these issues be effectively addressed through the proposed capability modularization? The current method section primarily explains the specific procedures, but offers very little discussion on the motivation behind these designs.
  4. First, the authors can refer to Weakness 2. Could the authors provide further analysis or empirical evidence demonstrating how Sensemaker alleviates these two limitations? I strongly suggest that the authors explain how their approach alleviates these two limitations with more discussion about design motivation, particularly process opacity.
6 comments
AuthorsOpenReview ↗

Response to Reviewer Ugis [Part 1/3]

We sincerely thank you for your detailed and insightful review of our paper. Below, we address your concerns point-by-point.

Design Motivation of Sensemaker (Weakness 1)

We thank the reviewer for recognizing the performance of Sensemaker and for raising this concern about its design motivation.

Drawing on the model of expert sensemaking discussed in Appendix A, Sensemaker organizes multi-document analysis into five capabilities with distinct responsibilities. FL locates relevant files, FP characterizes specific files, and CE extracts required content, supporting information acquisition and evidence construction in the foraging loop. DP processes collected data and EV checks whether claims are supported by cited sources, supporting evidence analysis and examination in the sensemaking loop. In particular, FL and FP serve different purposes: FL discovers candidate files across the collection, while FP provides reusable descriptions of file contents and roles to guide subsequent analysis.

We randomly sampled 100 failed execution trajectories from the baseline methods (20 samples from each of the five agent-based methods.) and manually counted occurrences of six error types: missing relevant files (E1), misjudging file attributes or roles (E2), content extraction errors (E3), data processing errors (E4), unsupported or misattributed claims (E5), and other errors (E6). These error types are not mutually exclusive: a single trajectory may contain multiple types of errors. As shown in the table below, the predominant errors in these trajectories concern the responsibilities explicitly modeled by the five capabilities. This provides empirical motivation for their selection.

Table. Failure analysis of existing methods.

Error TypeCountCorresponding Sensemaker capabilities
E163FL: Identify and retain candidates that warrant further exploration
E247FP: Accurately describe the file and its purpose
E339CE: Obtain accurate and traceable facts
E458DP: Properly process and integrate existing data
E531CE (information source binding) and EV (information source verification)
E68Other errors (tool execution, budget caps, task understanding, and so on)

To further examine the capability boundaries, we evaluated four additional variants: merging FL and FP; merging CE and DP; merging FL, FP, and CE; and distributing evidence verification across the other modules. As shown in the table below, all four achieve lower answer scores than Sensemaker. Together with the five module-removal variants reported in the paper, we can find that FL primarily improves file coverage, FP improves relevance discrimination, CE supports effective and efficient evidence extraction, DP improves downstream data processing, and EV improves answer quality at a modest token cost. These results support both explicit capability modeling and the chosen boundaries.

Table. Performance investigation.

VariantSco.Sco.Sco.(gold)Sco._{(gold)}RRPPT.Num.T.Num.
Sensemaker66.7±0.4477.5±0.3583.6±0.5777.0±0.510.405±0.008
merging FL and FP56.6±0.8775.2±0.6665.9±0.8362.8±0.710.484±0.014
merging CE and DP50.9±0.6969.7±0.8076.1±0.7268.4±0.600.692±0.018
merging FL, FP, and CE52.2±0.6167.4±0.9663.2±1.2456.9±1.110.737±0.020
moving EV into the others59.4±1.1573.1±0.8480.3±0.9572.2±1.260.951±0.023

We have added the analysis above in Appendix A in the revised paper.

Standard Deviations of Experimental Results (Weakness 3 & Question 1)

We agree that reporting standard deviations is important for assessing the stability and reliability of results. For Tables 2 and 4, we show partial results here. The complete results are provided in the revised paper. Compared with the agent-based baseline methods and variants, Sensemaker achieves the highest score with the lowest variation across runs, indicating that Sensemaker solves MDocAQA in a stable and effective manner.

Table 2. Experimental results with GLM-5.2.

MethodLoongxAFSFRAMESMuDA
LongAgent72.2${\pm1.21}$38.2pm1.49{\\pm1.49}31.8pm1.62{\\pm1.62}53.9{\\pm1.50}\
CoA71.2${\pm1.42}$36.7pm1.55{\\pm1.55}28.6pm1.77{\\pm1.77}51.8{\\pm1.72}\
MAIN-RAG61.3${\pm1.64}$43.9pm1.71{\\pm1.71}24.3pm2.09{\\pm2.09}59.0{\\pm1.86}\
BELLE73.2${\pm0.92}$49.7pm1.13{\\pm1.13}36.3pm1.48{\\pm1.48}65.1{\\pm1.37}\
LongTA74.7${\pm0.69}$53.9pm1.13{\\pm1.13}44.2pm1.44{\\pm1.44}62.0{\\pm1.13}\
Sensemaker (ours)79.9${\pm0.27}$67.0pm0.43{\\pm0.43}64.6pm0.70{\\pm0.70}72.3{\\pm0.65}\

Table 4. Performance investigation of Sensemaker.

VariantSco.Sco.
Sensemaker66.7${\pm0.44}$
w/o dedicated FL54.8${\pm0.81}$
w/o dedicated CE52.2${\pm0.73}$
w/o dedicated DP49.2${\pm0.79}$
w/o capability instantiation45.5${\pm0.94}$
w/ general subagent41.8${\pm1.05}$
w/o capability layer40.9${\pm1.52}$
AuthorsOpenReview ↗

Response to Reviewer Ugis [Part 2/3]

Methodological Differences and Unique Benefits (Question 2)

We appreciate your concern about methodological differences and unique benefits of Sensemaker.

(1) Differences from MuDA

MuDA is a fixed framework built for MuDABench, with components specialized for its document characteristics. It cannot be directly applied to other MDocAQA tasks. We agree that Sensemaker is more flexible than MuDA, but the fundamental difference lies in the framework design. Sensemaker models multi-document analysis through distinct capabilities based on the common characteristics of MDocAQA. This design provides generality across diverse MDocAQA tasks, flexibility to choose different solution paths, and controlled execution through five fixed capability modules.

The table below compares two methods on MuDABench. Since GPT-5.5, used for evaluation in the original paper (Appendix D.1), is no longer available, we use GPT-5.6 Sol instead. We can find that Sensemaker performs better even on the benchmark for which the MuDA workflow was developed.

Table. Comparison results.

ModelMethodScore ± Standard Deviation
Kimi-K2.6MuDA58.3±1.10958.3{\tiny \pm 1.109}
Sensemaker70.2±0.65170.2{\tiny \pm 0.651}
GLM-5.2MuDA62.9±0.93162.9{\tiny \pm 0.931}
Sensemaker72.4±0.61972.4{\tiny \pm 0.619}
Qwen3.5MuDA56.8±1.26256.8{\tiny \pm 1.262}
Sensemaker68.0±0.79168.0{\tiny \pm 0.791}

(2) Benefits over Existing Multi-agent Workflows

The key difference lies in the framework design. Existing MDocAQA multi-agent workflows typically decompose tasks to reduce the number of documents handled by each subtask. Sensemaker models the distinct capabilities required by MDocAQA to reduce the range of capabilities required within a single agent execution. This design offers three main advantages:

  • Reduced agent overload. Each module handles one responsibility in an independent session, reducing the need to coordinate different types of operations within a single execution.

  • Reduced process opacity. Actions and intermediate results within each trace serve one clear purpose, making the process easier to inspect and errors easier to locate.

  • Scalability through stable responsibilities. As document collections grow, coordinating multiple capabilities across more files becomes increasingly difficult for agents. However, since Sensemaker keeps each agent focused on one responsibility, handling more files mainly involves applying the same type of operation to a larger collection. This enables Sensemaker to handle larger document collections.

We have added the corresponding discussion in Appendix J of the revised paper to further clarify the methodological differences and advantages of Sensemaker.

AuthorsOpenReview ↗

Response to Reviewer Ugis [Part 3/3]

Discussion of Agent Overload and Process Opacity (Weakness 2 & Question 3/4)

We appreciate your questions about agent overload, process opacity, and their connection to performance gains. We first explain the causes of these issues and why capability modularization helps, then present additional experiments.

Underlying Causes and Design Motivation (Question 3)

The original manuscript discusses these issues and our design motivation in the Introduction and Related Work sections, while the Methodology section focuses on the framework and its procedures.

Here, agent overload concerns the burden of coordinating multiple capabilities and maintaining their intermediate states within one execution, rather than task complexity itself. Process opacity refers to difficulty inspecting intermediate results and identifying where errors arise in an execution trace.

The two issues share a common cause: different processing responsibilities and their intermediate states are coupled within a single execution. An agent must repeatedly switch between file location, relevance assessment, evidence extraction, and so on. Managing these different demands contributes to agent overload. Mixing operations with different purposes and their outputs in one trace makes intermediate results harder to inspect, contributing to process opacity.

Sensemaker models multi-document analysis at the capability level, separating dynamic task planning from modules with stable responsibilities. Each module call runs in an independent session with corresponding tools and explicit output requirements.

  • Agent overload: Each module in Sensemaker handles one capability, reducing the demands of switching between responsibilities and managing their intermediate states.

  • Process opacity: Each trace only contains actions and outputs of one clear responsibility, making processes clear and easy to review.

We have added this discussion to the Methodology section to clarify the design motivation and how capability modularization alleviates both issues.

Empirical Analysis (Weakness 2 & Question 4)

We provide further analysis and empirical evidence demonstrating how Sensemaker alleviates these two limitations.

(a) Agent Overload:

Under the definition of agent overload above, Sensemaker alleviates agent overload by decoupling capabilities, reducing the range of responsibilities an agent must coordinate within one execution.

We extend the performance investigation with four additional variants: merging FL and FP; merging CE and DP; merging FL, FP, and CE; and distributing evidence verification across the other modules. Alongside Sco.Sco., we report three metrics per agent execution: token number (TkNTkN), tool-call number (TcNTcN), and repeated exploration rate (RERRER). As the transition progresses from full coupling (Sensemaker w/ general subagent) to partial coupling (4 variants) and finally to complete decoupling (Sensemaker), Sco.Sco. increases significantly, while TkNTkN, TcNTcN, and RERRER show a decline, supporting the effectiveness of capability decoupling in alleviating agent overload.

Table 7. Performance Investigation.

VariantSco.Sco.TkNTkN (million tokens)TcNTcNRERRER (%)
Sensemaker66.70.0738.717.9
merging FL and FP56.60.13511.9012.1
merging CE and DP50.90.12711.7611.3
merging FL, FP, and CE52.20.15613.4816.8
moving EV into the others59.40.19314.2119.4
w/ general subagent41.80.24817.3421.6

(b) Process Opacity

As explained above, combining operations with different purposes and their intermediate results in one trace makes inspection and error localization difficult. Sensemaker organizes these operations into capability modules with distinct responsibilities and independent sessions. This makes intermediate results easier to examine.

To confirm this, we use GLM-5.2 as the backbone model and randomly sample 100 questions that all methods answered correctly. We manually insert errors into the corresponding execution traces and ask GLM-5.2 to locate them, measuring diagnosis success rate and average diagnosis time. Table 8 shows that Sensemaker achieves the highest success rate and the lowest average diagnosis time. These findings indicate that its traces are easier to inspect and its errors easier to locate, supporting the effectiveness of Sensemaker in alleviating process opacity.

Table 8. Experimental results of trajectory diagnosis.

MethodSuccess RateAverage Diagnosis Time (second)
LongAgent0.69957.02
CoA0.641024.31
MAIN-RAG0.72642.83
BELLE0.75612.94
LongTA0.79843.85
Sensemaker (ours)0.94371.26

We have added the supporting experimental evidence in Appendix I.

Reviewer_hd8wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Sensemaker uses a coordinator and five specialized agents to answer multi-document analytical questions. Each agent operates in an independent agent session with specialized tools and structured inputs and outputs. The coordinator selects the next capability and supplies the relevant information. Experiments across six benchmarks and three backbone models report improvements in answer quality and token efficiency.

Critical strengths
  1. The empirical evaluation is substantial, with consistent improvements across multiple benchmarks and models.
  2. The ablations are thorough and meaningful. In particular, the comparison against a single session retaining the same role prompts and tool permissions supports the value of separate specialist execution, while the general-subagent baseline tests whether generic delegation alone explains the gains, both showing the effective of the framework.
Critical weaknesses
  1. Methodological novelty appears incremental. The framework primarily combines predefined role prompts, tool restrictions, structured handoffs, and independent execution contexts within a coordinator–specialist architecture. The effectiveness is convincing, but the broader design seems arbitrary.
  2. What motivates the choice of these five capabilities? A systematic analysis of failures in existing methods would help justify the motivation of the current design.
Questions for authors
  1. What generalizable design insight does the framework establish?

  2. What motivates these five capabilities and their boundaries? Can the authors provide a systematic failure analysis of existing methods to justify this decomposition over simpler or alternative groupings?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hd8w [Part 1/2]

We sincerely appreciate your thorough review and constructive feedback. We have taken your comments seriously and addressed your main concerns with point-by-point responses below.

Methodological Novelty (Weakness 1)

We understand your concern about methodological novelty. However, the novelty of Sensemaker lies in that we propose the first general MDocAQA framework to model multi-document analysis at the capability level by decoupling and explicitly modeling 5 fundamental capabilities required for MDocAQA tasks.

A common problem in existing MDocAQA methods is that different processing responsibilities and their intermediate states remain coupled within a single execution. Coordinating file location, relevance assessment, evidence extraction, and other capabilities contributes to agent overload. Mixing these operations and their outputs makes intermediate results difficult to inspect and errors difficult to locate, contributing to process opacity. Even multi-agent task decomposition can retain this coupling, since each subtask may still require several capabilities. However, instead of relying on task-level decomposition, Sensemaker approaches MDocAQA from a new perspective centered on capabilities. Based on the characteristics of multi-document analysis, it identifies five fundamental capabilities and models the solution process as a flexible combination of these capabilities. This narrows capability demands within each module execution and gives each local trace a clear responsibility, alleviating both issues.

Role prompts, tool restrictions, structured handoffs, and independent execution contexts are common means of implementing agent frameworks. They provide the underlying support for Sensemaker. Our experiments show that variants using similar implementation mechanisms perform substantially worse when capability modeling and decoupling are weakened or removed. These comparisons support capability modeling and decoupling as the core architectural contribution and a key driver of the performance gains.

Generalizable Design Insight (Question 1)

The generalizability of Sensemaker stems from its architectural design: modeling multi-document analysis through stable capabilities while retaining dynamic task planning. This design offers three insights:

  • General-purpose capabilities. Inspired by the model of expert sensemaking, the five capabilities capture common information-processing needs in MDocAQA. These capabilities can be combined in different ways to address diverse multi-document analysis tasks.

  • Scalability through stable responsibilities. As document collections grow, coordinating multiple capabilities across more files becomes increasingly demanding. Sensemaker keeps each agent focused on one responsibility, so handling more files primarily expands the work within that responsibility without introducing additional types of processing demands. This supports handling larger document collections.

  • Flexibility with controlled execution. The task layer dynamically chooses and revises solution paths according to the question and intermediate results. Meanwhile, execution follows five fixed capability modules with clear responsibilities and output requirements, preserving control over how each operation is performed.

More broadly, Sensemaker suggests a design principle for agentic systems: keep execution responsibilities stable while allowing their composition to adapt to each task. This principle may also guide other complex workflows that combine recurring information-processing capabilities.

AuthorsOpenReview ↗

Response to Reviewer hd8w [Part 2/2]

Design Motivation of Sensemaker (Weakness 2 & Question 2)

We understand your concern about the design motivation and capability boundaries of Sensemaker.

Drawing on the model of expert sensemaking discussed in Appendix A, Sensemaker structures multi-document analysis around five capabilities with distinct responsibilities: FL identifies relevant files, FP characterizes specific files, CE extracts required content, DP processes data, and EV verifies evidence. Conceptually, FL, FP, and CE correspond to searching for files, assessing their relevance, and extracting evidence within the foraging loop. DP and EV support the sensemaking loop by transforming collected evidence and checking whether claims are supported by their cited sources.

To systematically examine failures in existing methods, we randomly sampled 100 failed execution trajectories from the baseline methods (20 samples from each of the five agent-based methods.) and manually counted occurrences of six error types. These include missing relevant files (E1), misjudging file attributes or roles (E2), content extraction errors (E3), data processing errors (E4), unsupported or misattributed claims (E5), and other errors (E6). These error types are not mutually exclusive: a single trajectory may contain multiple types of errors. As shown in the table below, the predominant errors in existing methods concern the responsibilities explicitly modeled by our five capabilities, providing empirical motivation for their selection.

Table. Failure analysis of existing methods.

Error TypeCountCorresponding Sensemaker capabilities
E163FL: Identify and retain candidates that warrant further exploration
E247FP: Accurately describe the file and its purpose
E339CE: Obtain accurate and traceable facts
E458DP: Properly process and integrate existing data
E531CE (information source binding) and EV (information source verification)
E68tool execution, budget caps, task understanding, and so on

To further examine the capability boundaries, we evaluated four additional variants: merging FL and FP; merging CE and DP; merging FL, FP, and CE; and distributing evidence verification across the other modules. As shown in Table 4, all four variants achieve lower answer scores than Sensemaker. Together with the five module-removal variants reported in the paper, these results support the benefits of dedicated capability modules and the chosen capability boundaries.

Table 4. Performance investigation of Sensemaker under different capability configurations.

VariantSco.Sco.Sco.(gold)Sco._{(gold)}RRPPT.Num.T.Num.
Sensemaker66.7±0.4477.5±0.3583.6±0.5777.0±0.510.405±0.008
merging FL and FP56.6±0.8775.2±0.6665.9±0.8362.8±0.710.484±0.014
merging CE and DP50.9±0.6969.7±0.8076.1±0.7268.4±0.600.692±0.018
merging FL, FP, and CE52.2±0.6167.4±0.9663.2±1.2456.9±1.110.737±0.020
moving EV into the others59.4±1.1573.1±0.8480.3±0.9572.2±1.260.951±0.023

We have added the analysis above in Appendix A in the revised paper.

Dear Reviewer hd8w,

Thank you again for your thoughtful feedback. We would like to follow up on our rebuttal and the revised manuscript. Please let us know if any concerns remain unresolved or if further clarification would be helpful. We would also appreciate it if you could revisit your assessment and rating in light of the clarifications and additional evidence we have provided.

Thank you for the time and effort you have devoted to reviewing our work.

Best regards,

All authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers

We sincerely thank all three reviewers for their careful reviews and constructive suggestions. We have revised the manuscript and marked the changes in YELLOW. Below, we summarize the main revisions and additional analyses. Detailed responses are provided in the replies to individual reviewers.

  • Methodological Novelty and Generalizable Design Insight (Reviewers hd8w, Ugis): We have expanded Section 3 and Appendix J to clarify the methodological novelty of Sensemaker: modeling multi-document analysis at the capability level through five modules with stable responsibilities while preserving dynamic task planning. The added discussion explains how this design alleviates agent overload and process opacity and supports generalization across tasks.

  • Motivation and Boundaries of the Five Capabilities (Reviewers hd8w, Ugis, x7K7): We have added failure analysis to Appendix A.2 (Table 5) to explain the selection of these five capabilities, showing that their responsibilities cover the predominant failures in 100 failed trajectories from baseline methods. We have also added four capability-repartitioning variants to Table 4. Together with the module-removal experiments, these results support dedicated capability modules and the chosen boundaries.

  • Agent Overload and Process Opacity (Reviewers Ugis, x7K7): We have expanded Section 3 to explain the causes of these issues and how capability separation helps. We have added execution analysis to Appendix I.1 (Table 7), showing fewer tokens, tool calls, and repeated explorations per agent execution. We have also added controlled error-diagnosis experiments to Appendix I.2 (Table 8), showing more accurate and faster diagnosis. These findings support reduced agent overload and improved process transparency.

  • Stability Across Repeated Runs (Reviewers Ugis, x7K7): We have added standard deviations over three runs alongside the mean results in Tables 2 and 4 of the revised paper. The results show that Sensemaker maintains high answer scores with low variation across runs.

  • Comparison with the MuDA Framework (Reviewer Ugis): We have provided additional comparisons on MuDABench using three backbone models in our response to Reviewer Ugis. The results show that Sensemaker achieves higher answer scores across all three backbone models.

  • Verification of File Recall and Precision (Reviewer x7K7): We have added a regular-expression-based consistency check for file recall and precision in our response to Reviewer x7K7. The two calculation strategies show close agreement.

General Reply on Methodological Novelty and Contributions

We understand the concerns about methodological novelty. We want to clarify that our novelty lies in that Sensemaker is the first general MDocAQA framework to model multi-document analysis at the capability level by decoupling and explicitly modeling 5 fundamental capabilities required for MDocAQA tasks. Existing MDocAQA methods often require an individual agent execution to coordinate several processing responsibilities. This coupling increases coordination demands and makes intermediate results difficult to inspect. Even multi-agent task decomposition can retain this coupling, since each subtask may still require several capabilities. However, instead of relying on task-level decomposition, Sensemaker approaches MDocAQA from a new perspective centered on capabilities. Based on the characteristics of multi-document analysis, it identifies five fundamental capabilities and models the solution process as a flexible combination of these capabilities. This design offers three advantages:

  • Reduced agent overload. Focused responsibilities reduce coordination demands within each execution.

  • Reduced process opacity. Actions and intermediate results within each trace serve one clear purpose.

  • Scalability. Scaling up work involving a single capability is simpler than scaling up work that requires coordinating multiple capabilities.

The selection of the five capabilities is supported by both design motivation and empirical analysis. Inspired by the model of expert sensemaking, they address file localization, file profiling, content extraction, data processing, and evidence verification. Our analysis of 100 failed trajectories shows that these responsibilities cover the predominant failures observed in baseline methods. Further module-removal and capability-repartitioning experiments show that weakening their explicit modeling or separation reduces performance. These findings support the coverage of the selected capabilities and the need to maintain their distinct responsibilities.

AuthorsOpenReview ↗

Summary of Rebuttal Updates and Reviewer Feedback

Dear Area Chair,

We thank you and the reviewers for the time you have devoted to our submission. To assist your assessment, we summarize the revisions we made in response to the reviews, and the outcome of the discussion.

All three reviewers (hd8w, Ugis, and x7K7) now give the paper a rating of 3 (Weak Accept), citing the strong empirical results of the paper and the value of the unified framework.

Below is a summary of the major updates and how we addressed the key concerns raised by the reviewers.

  • Methodological novelty and generalizable design insight (Reviewers hd8w and Ugis). We have expanded Section 3 and Appendix J to clarify the methodological novelty of Sensemaker: modeling multi-document analysis at the capability level through five modules with stable responsibilities while preserving dynamic task planning. This design limits the demands on each agent and makes intermediate results easier to inspect.

  • Rationale for the five capabilities and their boundaries (Reviewers hd8w, Ugis, and x7K7). We clarified how theory and empirical evidence guided the five-module design. The expert sensemaking model motivated the capability-based architecture, and recurring errors in the execution traces of baseline methods guided our choice of the five capabilities (Appendix A.2, Table 5). We then evaluated the resulting boundaries through module-removal experiments and comparisons with alternative capability partitions (Table 4).

  • Agent overload and process opacity (Reviewers Ugis and x7K7). First, we expanded Section 3 to explain why agent overload and process opacity arise in existing methods and how Sensemaker alleviates both. We added a workload analysis and a controlled error-localization experiment in Appendix I (Tables 7 and 8). Second, we examined how alleviating these issues contributes to better MDocAQA performance. In our follow-up to Reviewer Ugis, we extended the ablations in Section 4.3 (Table 4) by comparing variants with different degrees of capability modeling and decoupling.

  • Stability across repeated runs (Reviewers Ugis and x7K7). Tables 2 and 4 now report the mean and standard deviation over three runs.

  • Comparison with existing multi-agent frameworks (Reviewer Ugis). We clarified how Sensemaker dynamically combines reusable capabilities and how this differs from assigning work by file or by predefined stage. In our response to Reviewer Ugis, we also compared Sensemaker directly with the MuDA workflow on MuDABench using three backbone models.

  • Verification of file recall and precision (Reviewer x7K7). In our response to x7K7, we used regular expressions to extract the file paths cited in final answers and recomputed file recall and precision from those paths. The resulting values closely matched those obtained with the original calculation.

Current Consensus:

  • Reviewer hd8w. The reviewer raised their rating to 3 (Weak Accept), confirming that our response had addressed their concerns.

  • Reviewer Ugis. The reviewer maintained a score of 3 (Weak Accept), acknowledging stronger empirical support for the capability design and its practical benefits.

  • Reviewer x7K7. The reviewer gave an initial rating of 3 (Weak Accept), and we addressed the concerns about module boundaries, error localization, and the reliability of the evaluation.

We hope this summary helps you assess the contribution and locate the evidence added during the discussion. We have uploaded the revised manuscript with the changes highlighted in yellow.

Best regards,
The authors of Submission 21189

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.