acceptodds
Under review as a conference paper at ICLR 2027

Beyond Monolithic Architectures: Multi-Agent Search and Knowledge Optimization for Agentic Search

Abstract

Agentic search has emerged as a promising paradigm for complex information seeking by enabling Large Language Models (LLMs) to interleave reasoning with tool use. However, prevailing systems largely rely on monolithic agents, which suffer from structural bottlenecks such as unconstrained reasoning outputs that inflate trajectories, sparse outcome-level rewards that hinder credit assignment, and stochastic search noise that destabilize learning. To address these challenges, we propose M-ASK (Multi-Agent Search and Knowledge), a framework for multi-agent joint optimization in agentic search. M-ASK decomposes the search process into two complementary groups of trainable agents: Search Behavior Agents, which govern planning and search actions, and Knowledge Management Agents, which refine, compress, and maintain the evolving internal context. Crucially, this role decomposition is not introduced merely for modular orchestration; instead, M-ASK jointly optimizes these heterogeneous agents within a unified training framework, so that search behavior and knowledge management are learned as a coordinated multi-agent policy. To make such coordination effective, M-ASK employs turn-level rewards that provide fine-grained supervision for intermediate search decisions and knowledge updates. Experiments on multiple QA benchmarks demonstrate that M-ASK consistently outperforms strong baselines, yielding both higher answer accuracy and substantially improved training stability.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
multi-agent, agentic search

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.80 over 5 reviews, above 27% of the others.
Reviewer_5a65OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

M-ASK divides multi-turn search into five roles that share an LLM and communicate through a structured knowledge state. The roles are jointly trained with PPO, using absolute answer F1 and per-turn F1 improvements as rewards. Experiments on seven QA benchmarks report higher average F1 and more stable training than the compared baselines, with additional evaluations across Qwen model sizes and an exploratory web-search setting.

Critical strengths
  1. Useful knowledge management and intermediate supervision. Removing knowledge management or turn-level rewards reduces multi-hop average F1 by 2.86 and 15.23 points, respectively (Table 4), supporting their value within this framework.

  2. Improved performance and training stability. M-ASK achieves 50.09 average F1 versus 46.79 for the strongest overall baseline (Table 1). In ten runs, it records no collapses through 1,000 steps, versus nine for Search-R1 under the reported setup (Table 3).

Critical weaknesses
  1. Limited evidence for generality and statistical reliability. All evaluated backbones are Qwen models between 0.5B and 8B. Despite the separate ten-run stability study, Tables 1 and 4 lack uncertainty estimates and clear run counts. Gains over the strongest baselines are small on AmbigQA (+0.31) and Musique (+0.72), with no gains on PopQA or Bamboogle. Cross-family evaluation and uncertainty estimates would strengthen confidence in the findings.

  2. Missing end-to-end latency comparison. Appendix K reports token counts and throughput, but not average task completion time; scheduling also differs between methods. It remains unclear whether sequential agent calls make M-ASK slower per query than Search-R1. A matched comparison including retrieval and all agent calls would clarify whether the accuracy gains require additional waiting time.

Questions for authors
  1. How many independent runs underlie Tables 1 and 4, and how were checkpoints selected? Please provide available uncertainty estimates, especially for the small performance margins.

  2. Does M-ASK take longer per query than Search-R1 under matched hardware and serving conditions? Please report available mean end-to-end times, including retrieval and all agent calls, or clarify whether this requires new measurements.

Reviewer_mAX9OpenReview ↗
Rating
2
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes M-ASK, a framework that reframes agentic search from a monolithic single-model trajectory into a joint multi-agent optimization problem. The pipeline is decomposed into two cooperating groups, Search Behavior Agents and Knowledge Management Agents, trained jointly under a Dec-POMDP formulation. All roles share one LLM backbone via PPO, differentiated only by role prompts. Experiments on 7 QA benchmarks with Qwen2.5 M-ASK achieves competitive performance.

Critical strengths
  1. The identification of three entangled bottlenecks (unbounded output length, sparse reward, retrieval noise) is a genuinely useful diagnosis of why monolithic agentic search is unstable, and it directly motivates each design choice.
  2. M-ASK brings about 60% output-token reduction, making the method attractive beyond pure accuracy.
Critical weaknesses
  1. All experiments are conducted on Qwen-2.5. Authors should conduct experiments on other models.
  2. Authors should compare with more new works, instead of just Search-R1.
Questions for authors

Under matched search-call budgets, does M-ASK still outperform Search-R1, and by how much?

Reviewer_c9f1OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes M-ASK, a framework that jointly optimizes search behavior and knowledge management through five role-conditioned agents sharing a language model. It maintains a structured knowledge state and assigns turn-level rewards based on changes in answer F1. Experiments report improved QA performance, training stability, and output-token efficiency.

Critical strengths
  1. The framework addresses a relevant problem: coordinating search decisions with evidence filtering and knowledge maintenance. The structured state and explicit update operations provide a clear mechanism for refining accumulated information.

  2. The reported improvements span seven QA benchmarks and multiple model scales, suggesting practical value beyond a single dataset or model configuration.

  3. The paper includes component ablations, repeated-run stability experiments, and computational measurements, providing several perspectives on the method’s effectiveness.

Critical weaknesses
  1. Limited evidence for the long-horizon motivation. The paper motivates its approach through long trajectories and context accumulation, but Table 11 reports approximately two retrieval calls per query on HotpotQA. The length analysis primarily measures generated output rather than input-context growth. The exploratory 20-turn web-search experiment is a useful addition, but its actual interaction lengths and evaluation protocol are insufficiently specified. Stronger evidence is needed to establish the claimed benefits on long search trajectories.

  2. The contribution of role decomposition is not sufficiently isolated. M-ASK simultaneously introduces role specialization, structured knowledge management, and turn-level rewards. The ablations show that these components help within the proposed framework, but do not establish how much improvement specifically requires the role decomposition. A simpler or merged-role baseline using the same knowledge representation, rewards, and training budget would make the central contribution more convincing.

  3. The credit-assignment mechanism has limitations that need further analysis. Search, Summary, and Update receive the same marginal F1 reward, which localizes feedback to a turn without distinguishing individual role contributions. Moreover, necessary intermediate evidence may receive zero immediate reward if it does not yet improve the predicted answer. The paper should clarify how later gains provide credit to earlier search actions and examine cases where useful progress precedes an observable answer improvement.

Questions for authors

Please see above.

Reviewer_dDvYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors argue existing agentic search systems mostly rely on a single monolithic agent, which suffers from 3 problems: (i) unconstrained, verbose reasoning outputs that lengthen trajectories; (ii) sparse, outcome-only rewards that make multi-turn credit assignment difficult; and (iii) noisy retrieved content that keeps accumulating in the context. The authors claim that these problems jointly lead to suboptimal trajectories and training instability (collapse). The paper proposes M-ASK, which decomposes the search process into Search Behavior Agents (Planning, Search, Answer) and Knowledge Management Agents (Summary, Update). The agents share a structured knowledge state consisting of a set of sub-query–evidence pairs and a predicted answer, which the Update Agent revises in place or extends at each turn. All roles share a single LLM, distinguished by role-specific instructions, and are jointly trained with PPO under turn-level rewards. The Planning and Answer Agents receive absolute F1, while the Search, Summary, and Update Agents share the per-turn F1 gain (ΔF1) of the intermediate answer. On seven QA benchmarks using Wikipedia as the retrieval corpus, M-ASK is reported to achieve the highest average F1 (50.09) among the compared methods, with zero collapses across 10 training runs (compared with a 9/10 collapse rate for Search-R1).

Critical strengths

S1. I believe that training collapse is a real issue. Based on the reported results, M-ASK achieves the best overall average F1 among the compared methods on the seven QA benchmarks, with the clearest gains on multi-hop datasets such as HotpotQA. The reported advantage over Search-R1 on NQ and HotpotQA also holds across all evaluated backbone scales (0.5B to 7B), which, if it holds up, suggests that the approach remains useful when model capacity is limited. The reported training behavior is also encouraging: M-ASK shows no collapse in 10 independent runs (vs. 9/10 for Search-r1 and keeps per-call outputs short. According to Appendix K, it also uses fewer output tokens per query and less training compute than the monolithic baseline. Taken at face value, these results point to a practical advantage in stability and efficiency.

S2. The empirical study is broad, the paper includes a model-scale analysis, a repeated-run stability study, component ablations, a comparison of parameter-sharing strategies, a cost analysis, and an exploratory long-horizon web-search check. The appendices also provide the full role prompts, agent specifications, pseudocode, a worked example, an anonymized code package. In my view, the author did indeed attempt to provide a great deal of information.

Critical weaknesses

W1. Method relative to closely related work, and the source of the gains, are not sufficiently discussed. ΔF1 has the form of potential-based reward shaping: with γ=1, the per-turn rewards sum to F1(a_T) − F1(a_0) (a_0 being the Planning Agent’s initial prediction). I think, closely related turn-level rewards already exist, e.g.,:

TIPS (Xie et al., 2026) uses the same potential-difference structure, with log-likelihood instead of F1, is the same problems / goals (e.g., for problems: sparse rewards, credit assignment, training collapse). But TIPS appears only as a row in Table 6 without discussion? (e.g., Tips cite Ng et al. (1999)). The experiments cover three cells of an architecture × reward design:

  • Search-R1 (monolithic, outcome-only): 35.64 multi-hop Avg
  • w/o T-L Reward (multi-agent, outcome-only): 28.47
  • M-ASK (multi-agent, turn-level): 43.70. A monolithic agent trained with the same turn-level ΔF1 reward under the same setting isnt evaluated; the process-reward baselines in Tab 6use different reward signals, cover only NQ and HotpotQA, and report no stability results. As a result, the contribution of decomposition, the paper’s central claim, is not isolated for either accuracy or stability. The collapse rate of w/o T-L Reward isnt reported. Minor: the optimizer used for the Search-R1 baseline is not stated. Sec. 4.3 cites Deng et al. (2025), but which concerns GRPO collapse in Search-R1? While M-ASK is trained with PPO. Fig. 2(a) shows both F1 and EM curves for Search-R1, but it is not stated whether these are 2 reward variants or two metrics of the same runs, nor which one Tab 1 and 3 report. I think these things should all be clearly stated for a formal paper.

W2: independent contribution of role decomposition is vague I think, the claim is that multi-role decomposition with joint optimisation drives the gains in accuracy and stability. However, the experiments do not separate the effect of decomposition from that of the turn-level ΔF1 reward. In Tab 4, keeping the full multi-role architecture but replacing turn-level rewards with a global reward lowers the multi-hop average to 28.47. This is below the monolithic Search-R1 (35.64), which is also trained with an outcome-only reward; on HotpotQA the scores are 42.57 vs. 46.87. Decomposition’s benefit appears to depend on the dense reward. Sec 4.4.2 likewise attributes the gain to the combination of the 2. The ΔF1 reward does not inherently require role decomposition. For example, one could probe the intermediate answer after each retrieval step of a monolithic agent and compute the same increment. The paper may lacks something (as shown in W1) If it performs comparably (e.g., a monolithic agent trained with the same turn-level ΔF1 reward), I mean, the contribution is better framed as a turn-level reward design, and the “beyond monolithic architectures” framing in the title and abstract would need to be revised.

W3: Description of the reward is inconsistent with its definition, also, that credit assignment mechanism is unclear. Sec. 2 claims “distinct turn-level rewards” for the Search and Knowledge agents? But Eq. 3 and Alg. 1 show all 3 agents share the identical r_iter = ΔF1, so the reward does not distinguish between the Search agent and the Knowledge Management agents. As a reader, I would feel somewhat confused. You do not specify whether R_{t,i,k} in Eq. 5 is accumulated across turns or restricted to a single turn, nor the value of γ. Both cases are problematic, let me give possible examples: If returns are accumulated across turns with γ≈1, the return from turn t telescopes to F1(a_T) − F1(a_t), the outcome reward minus a baseline (undermining the “dense credit assignment” interpretation). Also, If returns are single-turn, bridging retrieval steps that do not change the current answer receive zero reward, which contradicts the multi-hop motivation. In the example in Appendix J, the prediction is Puebla both before and after Turns 2 and 3 (ΔF1 = 0), so verification, efinement behaviors it showcases arent rewarded. I suspect there may be other possibilities, but the current paper, at the very least, does not allow readers to be certain. I would like your authors clarify the points that I found confusing. Minor: there are also some inconsistencies in the subscripts used in Sec. 3.2.1 and Eq. 3 for the same turn-level reward (minor clarity issue for marks).

Questions for authors

Questions / requests: (a) Unlike TIPS, which adds the shaping term to an explicit outcome reward, the Search, Summary and Update agents receive only ΔF1. Invariance then holds only if returns accumulate across turns with γ = 1, in which case the return equals the outcome reward minus a state-dependent baseline, i.e., the same variance-reduction mechanism as TIPS. Under single-turn returns it does not hold. Which regime is used? (b) maybe adding a monolithic baseline trained with the same turn-level reward, and report the collapse rate of w/o T-L Reward, is needed?. minor: which optimizer was used for Search-R1 (C) Is R_{t,i,k} in Eq. 5 accumulated across turns or restricted to a single turn? What values of γ and λ are used? Could the authors report an ablation that removes only Eq. 3 (the shared ΔF1 reward) while keeping Eq. 2? Would the gain in the w/o T-L Reward ablation then be attributable to ΔF1, or to the per-turn F1 supervision of the Planning/Answer agents?

Reviewer_VAt1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes M-ASK, a multi-agent framework for jointly optimizing search behavior and knowledge management in agentic search systems. The system decomposes the search process into 5 specialized roles sharing a common LLM backbone: Planning, Search, Summary, Update, Answer. The central contribution is the joint optimization of these heterogeneous roles using PPO with parameter sharing and a hybrid reward mechanism. The Planning and Answer Agents receive absolute answer-level F1 rewards, while the Search, Summary, and Update Agents share a turn-level incremental reward defined as the change in answer F1 between consecutive search turns. M-ASK archives an average F1 of 50.09 on 7 QA benchmarks, outperforming the reported baselines. It also demonstrates substantial improvements in the reported training stability experiments.

Critical strengths
  • Clear and well-structured framework. The paper presents a clear multi-agent framework that decomposes agentic search into five specialized roles, providing a structured approach to handling different stages of the search process.
  • Consistent empirical improvements. M-ASK achieves consistent performance gains over Search-R1 across seven QA benchmarks, demonstrating the effectiveness of the proposed framework across different question-answering tasks.
Critical weaknesses
  • Limited methodological novelty. The overall role decomposition is relatively similar to prior work such as WebWeaver (arXiv:2509.13312).
  • Outdated experimental comparisons. The backbone model, baselines, and related work are relatively outdated, with no comparison against more recent methods and models released since January 2026. This makes it difficult to assess the competitiveness of M-ASK against the current state of the art.
  • Insufficient evidence for multi-agent credit assignment. The proposed turn-level reward mechanism does not convincingly establish precise credit assignment among different agents, since the same reward signal is shared across the Search, Summary, and Update agents.
Questions for authors
  • Attribution of turn-level rewards. How does the shared turn-level reward distinguish the individual contributions of the Search, Summary, and Update agents? Have the authors explored agent-specific reward designs or other approaches to validate the effectiveness of the proposed credit assignment mechanism?
  • Impact of role-specific prompts on training stability. To what extent does the reported training stability come from the specialized and detailed prompts designed for each role, rather than the multi-agent architecture itself? Would Search-R1 achieve comparable stability and performance if equipped with similarly detailed prompts and instructions?
  • Generalization to realistic web-search environments. How well does M-ASK generalize beyond the fixed Wikipedia retrieval setting to realistic web-search environments with dynamic content and heterogeneous sources? Can the authors provide more comprehensive experiments or clarify the limitations of the current evaluation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.