- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces the AsymComm benchmark, a multi-agent setting in which 2 LLMs are presented with disjoint evidence and must cooperate to answer the question. To promote communication between these agents, the authors test many different communication protocols which differ in how context is shared while working through the problem, such as Handover, Select, Escalate, etc. When testing out these protocols, the authors found that context handover results in decreased performance even with context lengths as short as 1000 tokens with the Qwen3-8B series. Other techniques introduced such as Guide perform better by sending only information about the context length and then proceeding with a different rule-based approach. Evaluation is performed on two existing QA benchmarks: HotPotQA and Musique.
Critical strengths
The paper tackles an important problem regarding agent collaboration and proposes some communication protocols to improve over naive context handover. There are many experiments and ablations performed to assess the efficiency and performance of these protocols on the two evaluation benchmark questions.
Critical weaknesses
My main concern with the paper is with its presentation. While the authors acknowledge that AI was used to polish the paper, in this case I found some sections hard to follow. For example, section 3.1 which introduces information topology and information profile seems a bit unnecessarily complicated to me to describe the idea of disjoint information. The topology notation does not seem to be used later in the paper. I also believe the naming choice for the protocols could be more clear, rather than simply using a single word such as Handover, Escalate, etc. The paper contains many figures highlighting key results from experiments, but I believe it would benefit from explanatory figures showing a real benchmark question and some snippets of real context emitted during the inference process.
Regarding the results, only one model size and family is tested. This is mentioned in the limitations, but is critical for improving the soundness of the results. I would also suggest benchmarking 2 different LLMs connected to harnesses such as Claude Code or Codex.
Questions for authors
See the weaknesses section for a summary of my concerns.
3 comments
Gentle Follow-up to Reviewer VfbP
Dear Reviewer VfbP, thank you again for your time and your thoughtful review. We would like to check whether our response and the revised paper have addressed your concerns. We would be glad to provide any further clarification before the discussion period closes. Thank you for your time.
Follow-up Before the Discussion Deadline
Dear Reviewer VfbP,
Thank you again for your time and thoughtful feedback. With only five hours remaining before the discussion deadline, we wanted to kindly follow up and ask whether you have had a chance to review our response and the revised manuscript.
We sincerely hope our revisions have addressed your concerns and would greatly appreciate hearing your thoughts.
Thank you again for your time and consideration.
Response to Reviewer VfbP
Thank you for the specific comments on the writing. We agree with them, and they led to the largest changes in this revision. We go through them in turn and end with the added models.
Section 3.1 is overly complex, and the topology notation is not used later.
We agree that the formalism there went beyond what the main text needs. The definitions the main text does not need, including the notation of the information profile and the details of its verification, are now in Appendix B. Section 3.1 keeps only an explanatory paragraph. It says that the evidence may be split between the two agents, held by one of them, held by both or held by neither, and that the agents do not know in advance which case they are in.
The one-word protocol names are unclear.
We adopted this suggestion, and it led us to reorganize the paper. Once the context is chunked, a protocol makes two decisions. The ranking decides which chunk goes first, and the schedule decides how many chunks go and when to stop. Every protocol is now named after these two choices, for example LLM-Rank then Escalate (Escalate in the submission) or LLM-Pick then Sender-Stop (Select in the submission). Table 1 in Appendix D defines one ranking or one schedule per row: who scores the chunks, what the model generates, what is sent in each round and who gives the stop signal. Section 5 introduces the rankings, then the schedules, then their combinations. The mapping to the old names is in the general response.
A figure with a real question and the actual messages exchanged is missing.
Section 5.3 now has such a figure (Figure 3). It walks through one real episode under LLM-Rank then Escalate, message by message. It is the first episode of the 7.2K set in which both agents answer wrongly after round 1 and correctly after round 2, and every text and number in it comes from the record of the run. The top shows the question and the two contexts, marks where the supporting chunk sits among the dozens of chunks on each side, and quotes the opening of each verbatim. The middle shows the ranking lists the two models actually wrote, the chunks and bits that crossed in each round, both answers after each round, and the round in which the exchange stopped because the answers no longer changed. The bottom sets the cost of this episode beside that of the same episode under Context Handover (Handover in the submission). Appendix M traces a real episode under four further protocols, and the new Figure 2 draws the three steps that every chunked protocol shares: chunking, ranking and scheduling.
Only one model family.
The scale axis now covers five models: Qwen3-8B, Llama-3.1-8B, Mistral-7B, Granite-3.3-8B and Phi-4-mini. Context Handover declines with size on all five, and at the largest size that fits its window each model has lost about half to three quarters of its accuracy at the natural size. LLM-Rank then Escalate declines more slowly and overtakes it within a few thousand tokens on every model. The accuracy threshold moves with the model, but in our tests so far reusing one model's threshold on another changes accuracy little. We also rebuilt the scale axis on a second QA dataset and tested long conversations (general response, Sections 6.1 and 6.4).
Testing two LLMs with a harness such as Claude Code or Codex.
Thank you for this suggestion. Our study targets edge devices with limited hardware, which is also why we focus on models of about 8B parameters, since larger models may not run well on such devices. The models behind harnesses such as Claude Code or Codex are very capable, but they are also very large, and the data-center hardware that runs them behaves quite differently from the edge devices we study. Because the two settings differ so much, this paper does not address that one. We do think it is a question worth studying, and we would like to explore it in future work.
On readability.
Beyond these points, the paper now follows one line of argument. Context Handover has a boundary, so the context must be chunked, and the two decisions left after chunking are the ranking and the schedule. Sections 5 and 6 follow this order. The sections before the experiments tell the story with few numbers, and each part of the experiments states its claim and the mechanism behind it before pointing to a figure or table. We hope the revision reads much more smoothly.