acceptodds
Under review as a conference paper at ICLR 2027

AsyncSwarm: Asynchronous Multi-Agent Training and Inference for Information Seeking

Abstract

Applications of large language models (LLMs) have expanded from single-turn question answering to single-agent systems using multi-step reasoning and tools. Multi-agent systems further extend their ability to solve complex tasks. A classic multi-agent system design consists of a lead agent that assigns work to parallel sub-agents. We refer to this system as an LLM swarm, which is useful for complex information-seeking tasks that require extensive search to gather and verify evidence from multiple sources. However, blocking inference makes the lead agent wait for the slowest sub-agent. During online reinforcement learning (RL) for LLM swarms, rollouts also wait for sub-agents to finish dozens of tool calls. We propose AsyncSwarm with two components to address these challenges: Asynchronous Swarm Inference lets the lead agent launch new sub-agents as soon as any sub-agent finishes, while Trace-Retrieval Swarm RL retrieves stored sub-agent responses to replace slow online execution during training. We use the open-source GLM-4.5-Air and GLM-4.7-Flash as the lead and sub-agent models, respectively, and apply RL gradient updates only to the lead model. Compared with blocking baselines, AsyncSwarm achieves 1.6× faster inference with similar performance and 5.8× higher RL training throughput. Under the same RL time budget, AsyncSwarm improves over the initial swarm by 10.6 points on WideSearch and 7.6 points on BrowseComp-ZH, compared with gains of only 1.4 and 1.3 points from online RL, respectively. These results show that AsyncSwarm enables more efficient training while achieving substantially better performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Agent, Multi-Agent, Agentic RL

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_eyi2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AsyncSwarm addresses delays in LLM swarms where a lead agent delegates information-seeking tasks to sub-agents. At inference time, the lead can act on a sub-agent’s response as soon as it arrives, without waiting for the others. During reinforcement learning, the method retrieves stored sub-agent responses instead of repeatedly running sub-agents, and updates only the lead model.

The authors evaluate the method on WideSearch, BrowseComp-ZH, and GAIA. On WideSearch, asynchronous inference is 1.62× faster than blocking inference with similar performance. Trace retrieval makes RL steps 5.84× faster than online RL, excluding the one-time cost of collecting traces. Under the same 22-hour RL budget, it yields larger performance gains than online RL on all three benchmarks. The evaluation uses real sub-agents at test time and includes analyses of retrieval and learned delegation behavior.

Critical strengths
  1. A key strength is that the lead agent learns from stored responses but improves on held-out tasks when evaluated with real sub-agents and live tools. This shows that the learned behavior carries over to real execution, which is the central concern with this training approach.

  2. The method also addresses a practical bottleneck in multi-agent reinforcement learning. Reusing a trace corpus makes training substantially faster without training the sub-agents or adding a complex retrieval system. Its simplicity and measured efficiency make it useful beyond the specific swarm studied here.

Critical weaknesses
  1. The RL comparison is not step-matched. Both methods train for 22 hours, but online RL completes roughly 39 GRPO steps (22 h at 33.49 min per step), while trace-retrieval RL completes 237. The larger gains may therefore come from doing more updates, rather than from better learning per update. A held-out evaluation of the trace-retrieval checkpoint around step 40, together with online RL reward curves if available, would help separate these explanations.

  2. A matched single-agent baseline is missing. The initial swarm scores 57.6 on WideSearch, below the reported 61.3 for its lead model alone, although those results use different settings. Please evaluate the single-agent lead with the same tools, context limit, and protocol to show whether the swarm itself helps before RL.

Questions for authors

Could you report held-out results for the trace-retrieval checkpoint at about step 40 and for the single-agent lead under the same setup?

2 comments

We appreciate your helpful comments on our evaluation.

W1 & Q1 — Evaluation after the same 40 steps

Your question separates learning at the same step count from the benefit of completing more updates. We compare both 40-step checkpoints on identical test questions, tools, and inference settings with real sub-agents.

Table A. Performance after 40 steps (%, differences in percentage points).

MethodRL stepsWideSearch Avg@4 Item-F1 (%)BC-ZH accuracy (%)
Initial policy057.5655.36
Online RL4059.056.7
Trace-Retrieval Swarm RL4060.7157.38
Trace-Retrieval Swarm RL − Online RL—+1.71+0.68

Both methods improve after 40 steps, with Trace-Retrieval Swarm RL scoring higher on both benchmarks. Useful policy learning is therefore already visible with steps matched. With 22 hours of RL (excluding collection), faster interaction permits more steps and gains of 9.2 / 6.3 percentage points over Online RL on WideSearch / BC-ZH (Table 1). We have included the 40-step results in Section 3.2 and Appendix H.1 (Table 9). The added comparison is highlighted in blue in the updated PDF.

W2 — Initial swarm performance and learned orchestration

We agree that the initial swarm score needs explanation: a swarm also needs an effective orchestration policy. Table A separates the external standalone result from our before/after-RL evaluations.

Table A. System scores (%); standalone and swarm results use different evaluation settings.

SystemResult sourceWideSearch Item-F1 (%)BC-ZH accuracy (%)
GLM-4.5-Air standalone agentExternal report61.353.1
Initial swarm (No RL)Our evaluation57.655.4
Swarm + Trace-Retrieval Swarm RLOur evaluation68.263.0

Within the same swarm architecture, sub-agent model, and evaluation settings, training only the lead agent raises WideSearch from 57.6 to 68.2 and BC-ZH from 55.4 to 63.0. This supports the value of learning orchestration within the same swarm.

The initial policy may not yet coordinate WideSearch's multi-entity searches well. Beyond finding individual facts, the lead agent must combine replies, identify missing entries, and arrange checks. Table B examines delegation and verification, which matter for this coverage requirement.

Table B. Changes in orchestration behavior on WideSearch (Appendix D).

BehaviorInitial swarmAfter RLChange
Trajectories with verification (%)87.2100.0+12.8 pp
Share of verification tasks (%)30.538.6+8.1 pp
Tasks per question7.378.50+1.13
Sub-agents per question6.282.60−3.68
Task-prompt length (tokens)254.8321.4+66.6

After RL, the lead agent assigns more tasks to fewer sub-agents, uses more verification, and gives longer prompts. These shifts are consistent with more follow-up work and checking of missing information, helping explain how the same swarm can improve through training its lead agent.

We have clarified repeated assignment in Section 3.4 and Appendix D.

Reviewer_iu2sOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes AsyncSwarm, a framework for improving the efficiency of multi-agent inference and RL training in information-seeking tasks. It introduces asynchronous sub-agent execution to reduce inference latency and a trace-retrieval-based RL approach that replaces expensive online sub-agent executions with previously collected responses. Experiments show substantial improvements in both training efficiency and downstream performance on several information-seeking benchmarks.

Critical strengths

The paper addresses a practical and important efficiency challenge in multi-agent systems, particularly the high cost of generating RL rollouts involving multiple sub-agents and tool calls. The proposed approach is simple and intuitive, and the experimental results demonstrate promising improvements in both inference speed and RL training efficiency. I also appreciate the analysis of how RL changes the lead agent's delegation and verification behaviors.

Critical weaknesses
  1. I am not fully convinced by the motivation for trading rollout quality for training efficiency. My understanding is that the proposed approach potentially sacrifices the quality of training rollouts, since retrieved sub-agent responses may be less accurate or appropriate than responses generated through actual execution, in exchange for faster RL training. However, I think many practical scenarios may prioritize achieving the best possible model performance during training, especially since training is usually a one-time cost, while inference costs can accumulate significantly during large-scale deployment. In addition, the applicability of the proposed method seems somewhat limited to knowledge-intensive information-seeking tasks. For more complex and diverse tasks, such as coding, it may be much harder to find previously cached sub-agent queries with sufficiently high similarity, making retrieval-based simulation less reliable. I think the paper could better motivate when and why this trade-off is desirable.

  2. I have some concerns about the design of the RL comparison experiments. First, I think the cost of collecting the retrieval corpus should be included when comparing the overall training efficiency of AsyncSwarm with online RL. Although the paper provides a theoretical amortized cost analysis, it would be helpful to report the actual end-to-end training cost, including the initial corpus construction. Second, the current experiments mainly compare RL performance under the same wall-clock training budget. I think an equally important comparison would be to evaluate both methods after the same number of RL update steps. This would help clarify whether AsyncSwarm sacrifices the quality of individual training updates in exchange for higher throughput, or whether retrieval-based rollouts might actually contribute to more stable or effective training. I believe this experiment is particularly important for understanding and positioning the paper's contribution.

  3. There seems to be a mismatch between the retrieval design and the reuse of sub-agents. Currently, retrieval uses only the sub-prompt as the key. However, the paper also observes that the trained lead agent becomes better at reusing existing sub-agents. When a sub-agent is reused, its response may depend not only on the current sub-prompt but also on its previous interaction history and context. In such cases, even an identical sub-prompt could correspond to completely different correct responses depending on the context. I think this potential mismatch deserves more attention. At the very least, it would be helpful to compare retrieval quality between initial sub-agent assignments and follow-up tasks assigned to previously used sub-agents.

  4. I think the current retrieval mechanism is still relatively basic, and there are several directions worth exploring. The proposed method completely replaces real sub-agent execution with retrieved responses during RL training, which raises two concerns. First, even when the best retrieval match has very low similarity, the system still returns its response, which could be incorrect or irrelevant. Second, as RL progresses, the lead agent may generate increasingly diverse sub-prompts that deviate from those collected by the initial policy, potentially leading to a more serious distribution shift (or policy drift). One possible improvement would be a hybrid approach combining retrieval with actual sub-agent execution: when retrieval confidence falls below a certain threshold, the system could fall back to a real rollout. The resulting responses could then be added to the retrieval corpus, gradually expanding its coverage and mitigating policy drift. Another concern is that the retrieval methods explored in the paper are relatively simple. High textual or semantic similarity does not necessarily imply that two sub-prompts have interchangeable answers, especially when they differ in critical details such as entities, dates, or constraints. It might be worth exploring more fine-grained matching methods, such as using a small LLM to judge whether a retrieved response is actually reusable for the current sub-prompt. Such methods could still be substantially cheaper than executing a full sub-agent rollout while potentially providing more reliable responses. I noticed that the paper also evaluates Dense Retrieval, but I could not find a clear specification of the embedding model used, which would be helpful for understanding and reproducing the results.

Questions for authors
  1. Regarding the mixed training data experiment, I wonder whether the improvement comes from task diversity or simply from having more unique training examples. The current comparison does not control for the total number of training samples: the mixed setting contains more unique questions than either the deep-only or wide-only setting. Therefore, it is difficult to tell whether the improvement comes from the complementary nature of the two task types or from the larger and more diverse training set. It would be helpful to include a comparison with a fixed number of training questions across different data mixtures. Also, could the authors clarify how many epochs or effective passes over the training data were performed for each configuration?

  2. There seems to be a mismatch between the RL training environment and the actual asynchronous inference process. Section 2.2 states that, during real asynchronous execution, wait returns the next completed sub-agent response. However, Algorithm 2 explicitly states that "During RL, pending tasks are handled in assignment order." This means the lead agent experiences a different response order during training and inference, which could affect its learned scheduling and coordination behavior. Have the authors considered recording the execution latency of each sub-agent task during corpus construction and using this information to simulate a more realistic asynchronous completion order during retrieval-based RL? I think this could help reduce the training-inference mismatch without introducing much additional overhead.

4 comments

Thank you for your careful reading and detailed suggestions.

W1 — Why accelerate RL training, and where does the method apply?

We agree that the quality of the deployed model is what matters. We aim to learn a better lead agent within a fixed training budget. Retrieval reduces repeated search and sub-agent work during RL; evaluation still uses real sub-agents. We have clarified this objective in the Introduction.

Imperfect retrieved feedback does not necessarily reduce policy quality. Although RL uses retrieved replies in place of real sub-agent execution, the lead agent still learns orchestration from final-answer rewards: when to search in parallel, follow up missing evidence, and verify. Figure 4 and Appendix D show increased parallel delegation and verification after training. Combined with higher test scores using real sub-agents, these changes support effective policy learning despite noisy feedback.

With matched initial models, hardware, and a 22-hour RL budget (excluding collection), Trace-Retrieval Swarm RL exceeds Online RL by 9.2 / 6.3 percentage points on WideSearch / BC-ZH, evaluated on held-out benchmarks with real sub-agents (Table 1). Thus, within the same RL budget, the method improves final policy quality while reducing training interaction cost.

We study information seeking, where stored replies supply facts and verification results. Code editing and testing depend on a changing environment state; stored replies cannot reproduce their effects, so we do not extend the claim to general coding (Limitations).

W2 & Q1 — Training cost, 40-step comparison, and data composition

Full training cost and corpus reuse (W2)

We agree that collection belongs in the full training cost. We also check performance at 40 steps, so efficiency is assessed through both time spent and the policy learned. Table A first reports collection, index construction, and RL.

Table A. Full timing for one run.

StageTimeTiming scope
WebQA collection9.36 hCollection job runtime; 128 concurrent requests
GISA collection3.30 hCollection job runtime; 64 concurrent requests
Main index construction1.31 sOne CPU
RL training22.00 h237 RL steps
Total for one run34.66 hFull collection cost and main index construction included

Including collection, one run takes approximately 34.66 hours. We have added this breakdown to Appendix H.2 (Tables 10–11).

Table B estimates the cost of the same number of updates under both methods, fixing measured Online RL / Trace-Retrieval Swarm RL mean GRPO step times at 33.49 / 5.73 minutes (5.84×). Each run has the workload of 22 hours of Trace-Retrieval Swarm RL. For MM runs sharing one corpus, the estimated speedup is S(M)≈(33.49/5.73)⋅22M12.66+22MS(M)\approx\frac{(33.49/5.73)\cdot 22M}{12.66+22M}, with collection charged once.

Table B. Estimated speedup with corpus reuse (22 hours of Trace-Retrieval Swarm RL per run).

Workload / first-run total1 run2 runs5 runs10 runsLimit
22.00 h RL / 34.66 h total3.71×4.54×5.24×5.53×5.84×

Sharing collection cost raises the estimated speedup from 3.71× for one run toward 5.84×. We have included these estimates in Appendix F.

Performance after the same 40 steps (W2)

The 40-step comparison checks learning at a common training length. Both 40-step checkpoints use the same test questions, tools, and inference settings, with real sub-agents at evaluation.

Table C. Performance after 40 steps (%, differences in percentage points).

MethodRL stepsWideSearch Avg@4 Item-F1 (%)BC-ZH accuracy (%)
Initial policy057.5655.36
Online RL4059.056.7
Trace-Retrieval Swarm RL4060.7157.38
Trace-Retrieval Swarm RL − Online RL—+1.71+0.68

At 40 steps, both methods improve over the initial policy, and Trace-Retrieval Swarm RL scores higher on both benchmarks. Retrieval thus supports useful learning with steps matched. Within 22 hours of RL (excluding collection), faster interaction allows more updates and larger gains of 9.2 / 6.3 percentage points on WideSearch / BC-ZH (Table 1). We have included the 40-step results in Appendix H.1 (Table 9).

Data composition and training exposure (Q1)

For Q1, the key control is task mixture at the same question count and training epochs. Table D makes the question-count difference in the original ablation explicit.

Table D. Original data ablation: questions and collection rollouts.

Training dataDeep questionsWide questionsUnique questionsCollection rollouts per question
Mixed3782536314
Deep-only37803784
Wide-only02532534

The four collection rollouts per question build the corpus; they are separate from RL training epochs. The original ablation (Table 3; revised Table 4) changes both task type and question count.

To isolate task composition, we fix 253 questions and four training epochs per run: 253 Wide-only, 253 downsampled Deep-only, or 126 wide + 127 deep. Other settings are unchanged.

Table E. Fixed question count and training epochs (%).

Training dataUnique questionsWideSearch Item-F1 (%)BC-ZH accuracy (%)GAIA accuracy (%)
Initial swarm (original No RL reference)—57.655.472.7
Wide-only25364.854.370.8
Deep-only25359.962.577.6
Mixed: 126 Wide + 127 Deep25366.061.576.9

With data size and epochs fixed, Mixed beats Wide-only on all benchmarks and improves WideSearch over Deep-only while largely retaining its BC-ZH and GAIA performance. This supports complementary benefits from the two task types, beyond having more questions. We have included this comparison in Appendix I.2 (Table 15).

W3 & Q2 — Context for each assignment and response ordering

Task context for each assignment (W3)

Thank you for pointing out the ambiguity in 'reuse'. We retain the sub-agent's identity and role, while each assignment starts with a fresh context. The previous task's dialogue is not carried over. The lead agent retains its own history and passes needed candidate answers, evidence, and constraints in a self-contained prompt. The sub-agent retains its search and tool-call history within the current assignment. We have clarified this boundary in Sections 2.1–2.2. These modifications are highlighted in blue in the updated PDF.

Response order during training and inference (Q2)

Training and inference share action semantics and state-update rules, but differ in reply order: by default, wait uses assignment order during Trace-Retrieval Swarm RL and completion order during asynchronous inference.

To test whether response order affects learning, we followed your suggestion to replay historical latency. We compare assignment order, completion order from random historical delays, and completion order from each retrieved reply's recorded latency. Only response order changes; initial weights, questions, seeds, corpus, retriever, RL settings, and update counts are fixed. All policies use asynchronous inference with real sub-agents and identical evaluation settings.

Table A. Training response-order comparison; evaluation uses asynchronous inference with real sub-agents (%).

Response order during RLWideSearch Item-F1 (%)BC-ZH accuracy (%)
Assignment order (original)65.6760.55
Random historical delays65.1860.24
Each reply's own historical delay65.7060.58

Using each reply's recorded delay changes both scores by only +0.03 percentage points; random delays slightly lower them. The assignment-order policy therefore already handles real completion order well in this setting. Matching the detailed historical timing adds little to the held-out performance.

Timing replay uses a virtual clock: wait advances directly to the next completion event, without waiting for the historical duration.

We have clarified response ordering in Section 2.3 and Algorithm 2, and repeated assignment to existing sub-agents in Section 3.4 and Appendix D. We have included the response-order comparison in Appendix I.4 (Table 17).

W4 — Retrieval reliability, coverage, and retriever choice

The fixed corpus always returns the highest-similarity reply. We therefore examine how much usable information remains at low similarity and whether the resulting policy transfers to real sub-agents.

We assess 200 real/retrieved pairs from 130 parent questions, with 50 initial GISA SFT subtasks per Word Jaccard bin. Same-source trajectories and identical prompts are excluded. Both full replies are automatically judged against the current request and reference answer for applicability and factual conflicts.

Table A. Retrieved reply applicability by similarity (%).

Word Jaccard binPairsRetrieved: at least partly applicableRetrieved: fully applicableFactual conflicts: real / retrieved
Q1: 0.1300–0.30935054.06.022.0 / 30.0
Q2: 0.3093–0.36255058.00.022.0 / 22.0
Q3: 0.3626–0.41535064.012.026.0 / 20.0
Q4: 0.4153–0.72905058.012.028.0 / 28.0
Overall20058.57.524.5 / 25.0

“Fully applicable” covers all requested information; “at least partly applicable” includes full coverage. Rates use all samples in each group; factual conflicts count as inapplicable.

In the lowest-similarity bin, 54% remain at least partly applicable and 30% have detected factual conflicts. These replies can support follow-up work when the lead agent checks their relevance and verifies possible errors. We have included this analysis in Appendix I.3 (Table 16).

For example, T195 requests leaders and birthdays for seven republics; the retrieved reply covers five. These facts can guide follow-up searches for the missing two.

For initial coverage, 631 questions (378 deep, 253 wide) × four independent rollouts yield 2,524 trajectories and 16,298 sub-prompt–reply pairs, capturing different decompositions and delegation choices (Section 3.1, Appendix A).

This initial collection covers different task decompositions, while the lead agent still generates new subtasks during RL. We therefore assess the fixed corpus by the policy it trains: higher held-out scores with real sub-agents (Table 1; Table B below) support transfer of orchestration. Figure 4 and Appendix D show changes in parallel work and verification.

For your fallback suggestion, the key is confidence that predicts reply reusability. An offline analysis retaining half the replies finds full applicability of 11.0% with similarity selection versus 12.5% expected with random selection (difference: 95% CI [−4.1, +1.1] percentage points). This does not show an advantage for lexical similarity alone. A possible extension is a small-model judge that checks entities, dates, and requested fields before choosing retrieval or real sub-agents, whose new replies could extend the corpus.

Policy performance with a different retriever

To test sensitivity to the training feedback, we replace only Word Jaccard with Dense (MiniLM) throughout RL, keeping the initial model, training questions, RL settings, 237 steps, and test set fixed. Evaluation uses real sub-agents.

Table B. Policy scores after 237 steps (%, differences in percentage points).

Training retrieverRL stepsWideSearch Item-F1 (%)BC-ZH accuracy (%)GAIA accuracy (%)
Word Jaccard23768.263.078.5
Dense (MiniLM)23768.463.778.8
Dense (MiniLM) − Word Jaccard—+0.2+0.7+0.3

Both retrievers yield policies that improve over the initial swarm with real sub-agents. The training benefit therefore persists with different retrieval feedback, slightly beyond Word Jaccard's lexical matching. We have included this comparison in Appendix I.1 (Table 14).

Dense uses frozen sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2, with 384-dimensional, L2-normalized vectors and cosine similarity between current and stored sub-prompts. We have specified this in Appendix B.

Original Table 4 (revised Table 5) changes inference-time retrieval at a fixed checkpoint; Table 14 changes the training retriever.

We have corrected Table 5 to cross-question mean ± SEM from one sample per question, with SEM=s/N\mathrm{SEM}=s/\sqrt{N}. N is each retriever's question count, and s the sample standard deviation across questions (Appendices C and E). Modifications are highlighted in blue in the updated PDF.

Reviewer_oD2POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AsyncSwarm, a framework for improving the inference and RL training efficiency of multi-agent systems on information-seeking tasks. It has two main components. First, asynchronous inference allows the lead agent to act on completed sub-agent responses without waiting for all sub-agents to finish. Second, Trace-Retrieval Swarm RL replaces online sub-agent execution during training with responses retrieved from a previously collected corpus of sub-task and answer pairs.

The authors train only the lead agent using GRPO, while keeping the sub-agent model fixed. Experiments on WideSearch, BrowseComp-ZH, and GAIA show improvements over the initial swarm and online RL under the same RL training time budget. The paper also reports a 1.62x inference speedup and a 5.84x reduction in average RL step time.

Critical strengths
  1. The paper addresses a practical efficiency problem in multi-agent systems. Waiting for slow sub-agents can increase execution time, and repeatedly running sub-agents during RL training is expensive. The proposed approach has a reasonable motivation, and the reported efficiency improvements are substantial.

  2. The experimental evaluation is relatively comprehensive. The authors compare different training settings, evaluate performance on multiple information-seeking benchmarks, and provide additional analysis of delegation and verification behavior. These experiments help demonstrate the practical effectiveness of the approach.

Critical weaknesses
  1. Limited novelty and insufficient analysis of asynchronous inference. The asynchronous inference component mainly adopts a straightforward scheduling strategy, allowing the lead agent to proceed when individual sub-agents finish. Similar asynchronous execution mechanisms already exist in several agent frameworks, making the methodological novelty limited. Moreover, the experiments primarily compare overall latency and answer quality, without examining whether acting on partial results may lead to missing useful information from unfinished sub-agents. In information-seeking tasks, different sub-tasks may provide complementary evidence, and proceeding before all results are available could potentially affect decision quality.

  2. Questionable reliability of Trace-Retrieval RL. The method replaces actual sub-agent execution with stored responses retrieved from similar sub-tasks. However, similarity between prompts does not guarantee that their answers are interchangeable. Even small differences in the requested information may lead to different correct answers. Inaccurate retrieved responses could therefore introduce misleading training signals and reinforce incorrect delegation decisions. Although Table 5 shows that retrieval-based rollouts achieve comparable final-answer quality on corpus-covered training questions, this does not adequately establish the reliability of the substitution for newly generated sub-tasks. Furthermore, the retriever ablation does not investigate how retrieval quality affects the final RL-trained policy.

  3. Incomplete evaluation of training efficiency. The reported 5.84x training speedup excludes the initial trace collection cost. Although Appendix F provides a theoretical analysis of amortized costs, no measured end-to-end training time comparison including data collection is provided. As a result, the actual efficiency gains of the complete training pipeline remain unclear. Additionally, the comparison with online RL under the same time budget does not isolate the effect of retrieval-based training from the benefit of performing more policy updates. A comparison under the same number of RL updates would provide stronger evidence.

Questions for authors
  1. Asynchronous inference: What distinguishes the proposed asynchronous inference mechanism from existing asynchronous multi-agent frameworks? How often does the lead agent finalize before all assigned sub-agents have returned, and does this lead to missing useful information? A more detailed analysis separating the latency gains from overlapping follow-up tasks and early termination of unfinished sub-agents would help clarify the actual benefits of the proposed mechanism.

  2. Reliability of Trace-Retrieval RL: How reliable are the retrieved responses when the current sub-task differs from the stored one? Could the authors compare retrieved responses with actual sub-agent outputs across different similarity levels? More importantly, how do retrieval errors affect RL rewards and the learned policy? An ablation examining the impact of retrieval quality on RL training would help validate this key assumption.

  3. Training efficiency: Could the authors report the actual end-to-end training time, including the initial trace collection cost, rather than only the time per RL step? Additionally, how does Trace-Retrieval RL compare with online RL under the same number of policy updates? This comparison would help distinguish the benefits of faster rollout generation from potential differences in training quality.

5 comments

Thank you for your thoughtful and constructive feedback.

W1 & Q1 — Asynchronous Swarm Inference, finalization, and latency savings

We agree that asynchronous inference is supported by existing frameworks such as AutoGen Core and AgentScope (Section 4). Our contribution is learning how the lead agent uses this interface: Trace-Retrieval Swarm RL trains when to delegate in parallel, wait for evidence, and finalize.

We would like to clarify why acting on one reply need not discard the others. Consider parallel searches A and B, with C verifying A's candidate answer:

Blocking inference: A and B both finish → start verification C.

Asynchronous inference: A's complete reply arrives → start verification C; B continues in parallel.

Collecting A's complete reply leaves B running. The lead agent can launch C and keep calling wait until B and C both return. Follow-up work can therefore begin without giving up B's evidence. Only finalize submits the answer and cancels unfinished tasks.

In the Ankang case added to Appendix H.4 (Table 13), C starts before B finishes and corrects the historical period of an industry description; the final answer uses this correction.

Early finalization and answer quality

Early finalization means calling finalize with at least one assigned reply uncollected. From the initial policy to RL step 199, its rate rises from 21.45% to 72.66% on BC-ZH (289 shared questions) and from 12.06% to 35.68% on WideSearch. After RL, subset / overall scores are 60.95% / 60.90% for BC-ZH accuracy and 62.89% / 67.57% for WideSearch Item-F1. Early finalization allows the lead agent to deliver answers sooner while achieving broadly similar answer quality (Appendix H.3, Table 12).

We then assess the overall speed–quality trade-off at the same checkpoint and sub-agent budget. Asynchronous / blocking inference achieves WideSearch Item-F1 of 68.2 / 66.9 in 14.72 / 23.81 minutes: 1.62× faster with a 1.3-percentage-point higher score (Table 2).

Latency savings

To check whether the speedup comes mainly from ending sooner, we first estimate the tail wait avoided by early finalization, then subtract it from the 545.7 seconds saved per question. We have included this breakdown in Section 3.2 (Table 3; calculation in Appendix H.5). These additions are highlighted in blue in the updated PDF.

Early finalization: 56.7 seconds per question (10.4%). Holding dispatched tasks and the generated answer fixed, we estimate the time to wait for all remaining tasks. Adding it back gives 15.66 minutes, versus 23.81 minutes for blocking inference.

Follow-up overlap (residual): 545.7−56.7=489.0545.7-56.7=489.0 seconds per question (89.6%). This residual estimate is consistent with overlap between follow-up work and ongoing tasks.

The resulting 8.62:1 ratio (about 9:1) means that most of the estimated saving remains after tail waiting is restored. This is consistent with the A/B/C mechanism: follow-up work overlaps with ongoing tasks even when the lead agent waits for all replies.

W2 & Q2 — Retrieval reliability and policy learning

Reply quality

Thank you for raising this concern. Similar prompts need not have interchangeable answers. We first check what information the retrieved replies retain, then test whether they train a policy that works with real sub-agents. The audit compares 200 real/retrieved reply pairs, with 50 initial GISA SFT subtasks per Word Jaccard bin, excluding same-source trajectories and identical prompts. An LLM judge assesses both full replies against the current request and reference answer for applicability and factual conflicts.

Table A. Reply quality by similarity (%); entries are real / retrieved.

Word Jaccard quartilePairsFully applicableAt least partly applicableFactual conflicts
Q1: 0.1300–0.30935010.0 / 6.062.0 / 54.022.0 / 30.0
Q2: 0.3093–0.36255010.0 / 0.064.0 / 58.022.0 / 22.0
Q3: 0.3626–0.41535022.0 / 12.062.0 / 64.026.0 / 20.0
Q4: 0.4153–0.72905028.0 / 12.068.0 / 58.028.0 / 28.0
Overall20017.5 / 7.564.0 / 58.524.5 / 25.0

“Fully applicable” covers all requested information; “at least partly applicable” includes both partial and full coverage.

58.5% of retrieved replies retain at least partly applicable information, but full coverage is less common than for real replies, while factual-conflict rates are similar. These replies provide facts to use and gaps to check. We have included this analysis in Appendix I.3 (Table 16).

What RL learns

Real search, including Google Search, also returns noisy results. The lead agent learns through RL how to use relevant information, identify missing evidence, and decide when parallel search or verification is needed. These are the orchestration decisions we aim to train.

Retrieved replies enter as environment observations. The RL loss covers only newly generated lead-agent tokens, and the reward scores the final answer to the full question. The policy is rewarded for solving the current question, rather than reproducing a stored reply. Errors can still mislead later decisions, so we test transfer with real sub-agents. With matched initial models, hardware, and a 22-hour RL budget (excluding collection), Trace-Retrieval Swarm RL exceeds Online RL by 9.2 / 6.3 percentage points on held-out WideSearch / BC-ZH (Table 1). Figure 4 and Appendix D show increased parallel delegation and verification.

Changing the training retriever

Table B tests the effect of retrieval choice on the learned policy. We train a separate policy with each retriever, changing only Word Jaccard to Dense (MiniLM) throughout RL. Initial model, training questions, RL settings, 237 steps, and test set are fixed; evaluation uses real sub-agents.

Table B. Policy scores after 237 steps with different training retrievers (%, differences in percentage points).

Training retrieverRL stepsWideSearch Item-F1 (%)BC-ZH accuracy (%)GAIA accuracy (%)
Word Jaccard23768.263.078.5
Dense (MiniLM)23768.463.778.8
Dense (MiniLM) − Word Jaccard—+0.2+0.7+0.3

Both policies improve over the initial swarm when evaluated with real sub-agents. The training benefit therefore persists under both lexical and dense retrieval, despite imperfect reply matches. We have included this comparison in Appendix I.1 (Table 14).

W3 & Q3 — Full training cost and performance at 40 steps

Thank you for pointing out these two comparisons. We address them separately: Table A includes collection in the full training cost, and Table B compares policy learning after the same 40 steps.

Table A. Cumulative training time from a fixed set of questions.

StageTime (h)Timing scopeResources
WebQA collection9.36-128 concurrent requests
GISA collection3.30-64 concurrent requests
RL: 237 steps22.00RL trainingOriginal training setup
Total training time34.66Collection, corpus construction, and RLFull collection cost charged to this run

One run totals approximately 34.66 hours: 12.66 hours of collection, 1.31 seconds of indexing, and 22.00 hours of RL. The 5.84× ratio compares mean GRPO step times during RL. We have included the full breakdown in Appendix H.2 (Tables 10–11).

The corpus can be reused for RL hyperparameter or retriever changes. Holding measured mean GRPO step times fixed at 33.49 / 5.73 minutes (Online RL / Trace-Retrieval Swarm RL), we estimate the cost for the same number of updates under both methods. Each run has the workload of 22 hours of Trace-Retrieval Swarm RL. For MM runs sharing one corpus, the estimated speedup is S(M)≈(33.49/5.73)⋅22M12.66+22MS(M)\approx\frac{(33.49/5.73)\cdot 22M}{12.66+22M}, with collection charged once. This gives 3.71× for one run, 5.24× for five, and approaches 5.84× as collection cost is shared across more runs (Appendix F).

Learning after the same 40 steps

Table B checks whether retrieval needs more updates to learn a useful policy. Both 40-step checkpoints use the same test questions, tools, and inference settings, with real sub-agents at evaluation.

Table B. Performance after 40 steps (%); Δ is the gain over the initial policy in percentage points.

MethodRL stepsWideSearchΔ from initialBC-ZHΔ from initial
Initial policy057.56—55.36—
Online RL4059.0+1.4456.7+1.34
Trace-Retrieval Swarm RL4060.71+3.1557.38+2.02
Trace-Retrieval Swarm RL − Online RL—+1.71—+0.68—

At 40 steps, both methods improve over the initial policy, and Trace-Retrieval Swarm RL scores higher on both benchmarks. Useful learning is already present with steps matched. Table 1 then shows the practical benefit under a fixed RL time budget, where faster interaction allows more updates. We have included the 40-step comparison in Section 3.2 and Appendix H.1 (Table 9).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.