acceptodds
Under review as a conference paper at ICLR 2027

AsymComm: Cost-aware Cooperative Communication Between On-device Agents Under Information Asymmetry

Abstract

As artificial intelligence and robotics advance, it is expected that on-device agents will be deployed in growing numbers, and that many situations will require these agents to cooperate. Such agents usually hold disjoint information, because each has observed a different aspect of the same phenomenon (place, event, etc.). The information between them is therefore asymmetric, and they must communicate to act together. We study how two such agents should communicate, through question answering, and introduce the AsymComm benchmark: two agents running the same model hold disjoint evidence, receive one question to solve together, and must reach a consensus answer, while every bit transmitted, every token read or generated and every round of communication is charged with the time it would cost on an edge device. The most direct protocol is context handover, in which one agent transmits its entire memory to the peer. However, we find that handover has a boundary. As the joint context grows, its accuracy declines well before the model's window is full, while its latency grows. We propose protocols for these constraints. Escalate, a protocol that sends ranked paragraphs in growing batches, is twice as accurate as handover at the largest size at a quarter of its latency. The boundary and a deadline on the answer combine into a rule for choosing a protocol. The rule itself forms a switching protocol, Guide, in which the two agents exchange only the sizes of their contexts and then run the protocol the rule selects; Guide obtains the accuracy of the best protocol at every context size, and because the rule is derived from the benchmark's measurements, a new protocol measured on the same axes enters it where it wins. The benchmark and the code that reproduces every experiment are released.

Primary area:
datasets and benchmarks
Keywords:
Multi-agent Systems, Information asymmetry, Edge AI, Agent Communication, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VfbPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces the AsymComm benchmark, a multi-agent setting in which 2 LLMs are presented with disjoint evidence and must cooperate to answer the question. To promote communication between these agents, the authors test many different communication protocols which differ in how context is shared while working through the problem, such as Handover, Select, Escalate, etc. When testing out these protocols, the authors found that context handover results in decreased performance even with context lengths as short as 1000 tokens with the Qwen3-8B series. Other techniques introduced such as Guide perform better by sending only information about the context length and then proceeding with a different rule-based approach. Evaluation is performed on two existing QA benchmarks: HotPotQA and Musique.

Critical strengths

The paper tackles an important problem regarding agent collaboration and proposes some communication protocols to improve over naive context handover. There are many experiments and ablations performed to assess the efficiency and performance of these protocols on the two evaluation benchmark questions.

Critical weaknesses

My main concern with the paper is with its presentation. While the authors acknowledge that AI was used to polish the paper, in this case I found some sections hard to follow. For example, section 3.1 which introduces information topology and information profile seems a bit unnecessarily complicated to me to describe the idea of disjoint information. The topology notation does not seem to be used later in the paper. I also believe the naming choice for the protocols could be more clear, rather than simply using a single word such as Handover, Escalate, etc. The paper contains many figures highlighting key results from experiments, but I believe it would benefit from explanatory figures showing a real benchmark question and some snippets of real context emitted during the inference process.

Regarding the results, only one model size and family is tested. This is mentioned in the limitations, but is critical for improving the soundness of the results. I would also suggest benchmarking 2 different LLMs connected to harnesses such as Claude Code or Codex.

Questions for authors

See the weaknesses section for a summary of my concerns.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VfbP

Thank you for the specific comments on the writing. We agree with them, and they led to the largest changes in this revision. We go through them in turn and end with the added models.

Section 3.1 is overly complex, and the topology notation is not used later.

We agree that the formalism there went beyond what the main text needs. The definitions the main text does not need, including the notation of the information profile and the details of its verification, are now in Appendix B. Section 3.1 keeps only an explanatory paragraph. It says that the evidence may be split between the two agents, held by one of them, held by both or held by neither, and that the agents do not know in advance which case they are in.

The one-word protocol names are unclear.

We adopted this suggestion, and it led us to reorganize the paper. Once the context is chunked, a protocol makes two decisions. The ranking decides which chunk goes first, and the schedule decides how many chunks go and when to stop. Every protocol is now named after these two choices, for example LLM-Rank then Escalate (Escalate in the submission) or LLM-Pick then Sender-Stop (Select in the submission). Table 1 in Appendix D defines one ranking or one schedule per row: who scores the chunks, what the model generates, what is sent in each round and who gives the stop signal. Section 5 introduces the rankings, then the schedules, then their combinations. The mapping to the old names is in the general response.

A figure with a real question and the actual messages exchanged is missing.

Section 5.3 now has such a figure (Figure 3). It walks through one real episode under LLM-Rank then Escalate, message by message. It is the first episode of the 7.2K set in which both agents answer wrongly after round 1 and correctly after round 2, and every text and number in it comes from the record of the run. The top shows the question and the two contexts, marks where the supporting chunk sits among the dozens of chunks on each side, and quotes the opening of each verbatim. The middle shows the ranking lists the two models actually wrote, the chunks and bits that crossed in each round, both answers after each round, and the round in which the exchange stopped because the answers no longer changed. The bottom sets the cost of this episode beside that of the same episode under Context Handover (Handover in the submission). Appendix M traces a real episode under four further protocols, and the new Figure 2 draws the three steps that every chunked protocol shares: chunking, ranking and scheduling.

Only one model family.

The scale axis now covers five models: Qwen3-8B, Llama-3.1-8B, Mistral-7B, Granite-3.3-8B and Phi-4-mini. Context Handover declines with size on all five, and at the largest size that fits its window each model has lost about half to three quarters of its accuracy at the natural size. LLM-Rank then Escalate declines more slowly and overtakes it within a few thousand tokens on every model. The accuracy threshold moves with the model, but in our tests so far reusing one model's threshold on another changes accuracy little. We also rebuilt the scale axis on a second QA dataset and tested long conversations (general response, Sections 6.1 and 6.4).

Testing two LLMs with a harness such as Claude Code or Codex.

Thank you for this suggestion. Our study targets edge devices with limited hardware, which is also why we focus on models of about 8B parameters, since larger models may not run well on such devices. The models behind harnesses such as Claude Code or Codex are very capable, but they are also very large, and the data-center hardware that runs them behaves quite differently from the edge devices we study. Because the two settings differ so much, this paper does not address that one. We do think it is a question worth studying, and we would like to explore it in future work.

On readability.

Beyond these points, the paper now follows one line of argument. Context Handover has a boundary, so the context must be chunked, and the two decisions left after chunking are the ranking and the schedule. Sections 5 and 6 follow this order. The sections before the experiments tell the story with few numbers, and each part of the experiments states its claim and the mechanism behind it before pointing to a figure or table. We hope the revision reads much more smoothly.

AuthorsOpenReview ↗

Gentle Follow-up to Reviewer VfbP

Dear Reviewer VfbP, thank you again for your time and your thoughtful review. We would like to check whether our response and the revised paper have addressed your concerns. We would be glad to provide any further clarification before the discussion period closes. Thank you for your time.

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer VfbP,

Thank you again for your time and thoughtful feedback. With only five hours remaining before the discussion deadline, we wanted to kindly follow up and ask whether you have had a chance to review our response and the revised manuscript.

We sincerely hope our revisions have addressed your concerns and would greatly appreciate hearing your thoughts.

Thank you again for your time and consideration.

Reviewer_7378OpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

∙\bullet The paper studies how two on-device agents should communicate when each has different private context.

∙\bullet The task is question answering where neither agent may have enough evidence alone, so they need to exchange information.

∙\bullet The authors introduce AsymComm, a benchmark that measures accuracy as well as device-style costs, such as reading tokens, generating tokens, wire bits, and communication rounds.

∙\bullet They show that simply sending all context, called handover, only works well for small contexts. As the context grows, it becomes slower and accuracy drops before the model's context window is full.

∙\bullet They propose a protocol Esalate, which sends ranked paragraphs in growing batches until the answers stop changing.

∙\bullet They also propose Guide, a simple rule that chooses the best protocol based on context size and deadline.

∙\bullet Validation is mainly done through experiments with Qwen3-8B on HotpotQA and MuSiQue-style splits, using simulated Jetson device cost models.

Critical strengths

∙\bullet The paper focuses on a practical issue: communication between small on-device agents under asymmetric information. I found the cost-aware framing important because many multi-agent papers ignore the real cost of reading, writing, and waiting.

∙\bullet The paper shows that sending everything is not always best, even before the model hits its context limit. This is a useful result, and the paired evaluation across context sizes makes the claim more convincing.

Critical weaknesses

∙\bullet Most main results rely on one model family and simulated device costs from published rates, not real edge-device runs. This matters because the main claims are about device-aware communication, so real-device measurements or more model diversity would make the conclusions stronger.

But it is understandable that working on real systems might not be feasible immediately. The contribution is still significant. I would encourage the authors to take up the real edge device implementations on future works.

∙\bullet Escalate works well mainly when the model can rank useful paragraphs reasonably. The paper shows this is the remaining bottleneck, but it makes me wonder how robust the method is on messier real-world memory, not just QA paragraphs.

Questions for authors

∙\bullet How sensitive is Guide to the exact measured thresholds? If the deployment model or device differs slightly, does Guide need to be fully re-benchmarked?

∙\bullet How would Escalate handle unstructured agent memories, where there may not be clean paragraph boundaries or obvious units to rank?

∙\bullet It would be helpful to test more model sizes/families to see whether the handover boundary is a general effect or mostly specific to Qwen3-8B. If it seems infeasible in the time constraint, I would encourage the authors to take it up as a future task. But if it is feasible, at least an analysis discussion would be great.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 7378

Thank you for your positive view of the cost-aware framing, and for stating your concerns so concretely. Your three questions map onto three sets of new experiments, so we answer them in turn.

On the single model family and the simulated device costs.

We repeated the scale axis on four more models, Llama-3.1-8B, Mistral-7B, Granite-3.3-8B and Phi-4-mini, with the same episodes and sizes as Qwen3-8B (Sections 6.1 and 6.4, Figure 5). Context Handover (Handover in the submission) declines with size on all five. LLM-Rank then Escalate (Escalate in the submission) declines more slowly and overtakes it within a few thousand tokens on every model. We therefore believe the boundary is not specific to one model.

The accuracy threshold S∗S^\ast does move with the model. It is about 2.5K tokens for Qwen, the highest of the five, and about 1.1K for Llama. For Mistral, Granite and Phi, it lies at or below the smallest size we measured, so Context Handover keeps only its speed at small sizes.

We agree that measurement on physical devices matters. We were not able to run this experiment for the revision, so we leave it to future work and state this in the limitations. Device rates move only the latency threshold, which has a closed form, so new rates give the new threshold directly. The accuracy ranking and the accuracy threshold do not depend on the rates.

On whether Escalate depends on the ranking, and whether it is robust on memory without clean paragraph boundaries.

The dependence on the ranking is real, and the revision now measures it directly (Section 6.2, Table 2 in Appendix E). We fix the schedule to Escalate and vary only the ranking. Under an oracle ranking, Escalate at 31.6K tokens is as accurate as Context Handover at the natural size, so the ranking sets the ceiling. The model's one-shot ranking decays with size, because ranking requires reading a long shard. Two rankings adapt to what has been received, and they come from Select and Request. LLM-Pick picks again every round and recovers about half to three quarters of the gap to the oracle. Peer-Request lets the receiver say in one line what it still lacks. The schedule compensates from the other side. Under the model's own ranking, Escalate is 6.5 to 18 points above a fixed Top-4 from the natural size on, because it keeps sending until both receivers' answers stop changing (Section 6.3).

For context without paragraph structure we ran LoCoMo, since the protocol only needs units it can enumerate. LoCoMo is a benchmark of very long-term conversations between two speakers. Each of its ten conversations runs over 19 to 32 dated sessions and more than twenty thousand tokens, and its questions ask about events and dates that may lie many sessions apart, with the evidence turns marked. We deal the odd sessions to one agent and the even sessions to the other, keep the 233 questions whose evidence spans both sides, and score token F1 because the answers are free-form. A unit is a whole session or a window of six turns. On both models we tested, LLM-Rank then Escalate matches or slightly exceeds the F1 of Context Handover, so ranking works on conversational units as well. The saving depends on the unit. Whole sessions are large, so a few of them already make up much of a shard and little is saved. With six-turn windows the protocol takes roughly half the latency of Context Handover at no loss in F1, and replacing its ranking with any of the three retrievers changes F1 by at most 2.5 points. The one setting to choose is therefore the size of the unit (Section 6.4, Table 16 in Appendix I).

On the sensitivity of Size-and-Deadline Routing (Guide in the submission) to its threshold, and whether a new model or device requires rerunning the benchmark.

In our tests so far the effect is small. Every protocol is measured at every size on the same episodes, so routing with any threshold can be composed offline, and the composed result is within one point of the routing we ran online. For three protocols beyond the boundary, any threshold up to 1.2×1.2\times the measured one keeps the mean accuracy along the scale axis within one point of the best. A threshold set too low only costs latency, and one set at 2×2\times the measured value loses about 2 points at most. Near the threshold the two sides are close in accuracy, so choosing the wrong side there costs little.

We also reused Qwen's threshold on the other four models. Against each model's best threshold, the loss in mean accuracy is under 1.5 points. A new model's best threshold can still be found by measuring two protocols on the scale axis, far less than the full benchmark. For a new device only the latency threshold moves, and it has a closed form in the two rates (Section 6.5, Appendices H and K).

AuthorsOpenReview ↗

Gentle Follow-up to Reviewer 7378

Dear Reviewer 7378, thank you again for your time and your thoughtful review. We would like to check whether our response and the revised paper have addressed your concerns. We would be glad to provide any further clarification before the discussion period closes. Thank you for your time.

AuthorsOpenReview ↗

Thank You for Your Feedback

Thank you for taking the time to revisit our work and for your encouraging feedback. We sincerely appreciate your constructive suggestions, which motivated us to strengthen the paper through additional experiments and analysis. We are glad that the revisions have addressed your concerns, and we greatly value your thoughtful engagement throughout the review process.

Reviewer_8ufMOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how two on-device LLM agents with different private knowledge communicate to agree on an answer. They primarily measure latency for edge hardware and evaluate on multi-hop QA tasks. They design a communication protocol that relies on simple heuristics for choosing based on context size.

Critical strengths

The problem formulation is understudied and could be applicable for multi agent systems.

Critical weaknesses

Missing simple RAG/BM25 baselines and only one model is evaluated. Latency is only simulated but not measured on real data.

Questions for authors
  • What is the learning or representation contribution, and why is ICLR the right venue?
  • Has the latency model been validated on real hardware?
  • Do the findings hold with other models and datasets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 8ufM

Thank you for your comments. We answer them point by point.

Missing simple RAG / BM25 baselines.

Thank you for raising this. We ran new experiments that add retrievers as rankings: BM25-Rank, Dense-Rank (BGE-base, 110M parameters) and Hybrid-Rank, which fuses the two orders. Here the context to be searched is on the other device, so the agent that holds it retrieves with the question and sends its top kk chunks in one round, and both agents answer from the union. This retriever ranking with Top-kk is the protocol in our setting closest to retrieval-augmented generation. We measured the retrievers on all five models, on 2WikiMultiHopQA, MuSiQue and LoCoMo, and on the evidence-density axis. Retrievers and the protocols of the submission thus share one design space (Figure 4, Table 2 in Appendix E, Sections 6.2 to 6.4).

A retriever scores each chunk separately and never reads the whole context, so it loses far less accuracy with size than Context Handover (Handover in the submission). BM25-Rank then Top-4 needs no extra component, but it trails LLM-Rank then Escalate (Escalate in the submission) on all five models and draws level only at the largest size on Qwen and at one size on Llama. With Qwen on HotpotQA, Dense-Rank then Top-4 overtakes LLM-Rank then Escalate from about 3K tokens on, at a fraction of its latency. Dense-Rank then Escalate adds the stopping rule of the submission to the dense ranking and is the most accurate deployable protocol from about 4K tokens on. Its cost is a small encoder and an index of the context on each agent, and an agent without them can rank with its own model.

The paper also states the qualifications. The encoder was trained on public retrieval data that include Wikipedia question answering, so HotpotQA is in-domain for it. On 2WikiMultiHopQA its lead is smaller, and under Escalate the three retrievers are within 5 points of one another. On LoCoMo, Hybrid-Rank is the best retriever under Top-4, and under Escalate all three retrievers are within 2.5 F1 points of LLM-Rank. On Granite-3.3-8B and Phi-4-mini, Dense-Rank then Top-4 is significantly more accurate than LLM-Rank then Escalate only at the largest size. No protocol is the best everywhere, so the paper reports every pair of ranking and schedule.

Only one model.

The scale axis now covers five models: Qwen3-8B, Llama-3.1-8B, Mistral-7B, Granite-3.3-8B and Phi-4-mini (Sections 6.1 and 6.4, Figure 5). All sixteen protocols of the design space were measured on every model, and Escalate helps the weakest ranking most on all five (Appendix H). The accuracy threshold moves with the model, but in our tests so far reusing Qwen's threshold on the other four models costs less than 1.5 points of mean accuracy.

Latency is only simulated. Has it been validated on real hardware?

Not yet. All latencies in the paper are computed from published device rates by declared formulas. We were not able to measure on physical devices for this revision, so we leave it to future work and state this in the limitations. Device rates move only the latency threshold, which has a closed form, so new rates give the new threshold directly. The accuracy ranking and the accuracy threshold do not depend on the rates.

Do the conclusions hold on other models and datasets?

The boundary of Context Handover holds on all five models and on 2WikiMultiHopQA, where the rebuilt scale axis shows the same shapes. On the long LoCoMo conversations, LLM-Rank then Escalate keeps the F1 of Context Handover, and with six-turn windows as units it takes roughly half the latency. Which protocol is the best changes with the model and the dataset (Section 6.4).

What is the learning or representation contribution, and why ICLR?

The ICLR 2027 call for papers lists datasets and benchmarks among its topics, together with applications to language and robotics, and ICLR regularly publishes work on LLM agents. The paper makes three contributions. The first is a benchmark. To our knowledge, it is the first to price communication protocols between two on-device agents by both accuracy and the time spent reading, writing and waiting on the device, with every comparison paired by episode. The second is an empirical finding about the long-context behavior of language models. Handing the context to the other agent is the most natural strategy between two agents, and it has a boundary that now holds on five models and two QA datasets. The third is the decomposition into ranking and schedule. It places retriever rankings, model rankings and handover in one design space and yields three findings: the ranking sets the ceiling, the receivers' own answers are a useful stop signal when the ranking is imperfect, and no protocol is the best everywhere. Size-and-Deadline Routing (Guide in the submission) follows from these findings and runs the protocol that answers the most episodes correctly within the deadline.

AuthorsOpenReview ↗

Gentle Follow-up to Reviewer 8ufM

Dear Reviewer 8ufM, thank you again for your time and your thoughtful review. We would like to check whether our response and the revised paper have addressed your concerns. We would be glad to provide any further clarification before the discussion period closes. Thank you for your time.

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer 8ufM,

Thank you again for your time and thoughtful feedback. With only 5 hours remaining before the discussion deadline, we wanted to kindly follow up and ask whether you have had a chance to review our response and the revised manuscript.

We sincerely hope our revisions have addressed your concerns and would greatly appreciate hearing your thoughts.

Thank you again for your time and consideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank the three reviewers for reading the paper closely and for their constructive comments. The reviews asked whether the conclusions hold on other models and datasets, and raised retrieval baselines, the sensitivity of the threshold, memory without paragraph boundaries and the names of the protocols. We ran new experiments for these questions and renamed the protocols. Together these changed the line of argument, so we describe it first. The revised paper has been uploaded.

A new line of argument. The main change in the revision is the line of argument of the paper, which now runs as follows. Context Handover has a boundary, so beyond it the context has to be sent in chunks. Once the context is chunked, a protocol makes two decisions. The ranking decides which chunk goes first, and the schedule decides how many chunks go and when to stop. Sections 5 and 6 follow this order, and every protocol is now named "ranking then schedule", so its name states what it does, as Reviewer VfbP asked. The retrieval baselines that Reviewer 8ufM mentioned enter the benchmark as one more ranking. The table maps the new names to the submission.

SubmissionRevisionWhat it does
HandoverContext HandoverSends the whole context
Read-aloudGenerated HandoverThe same content, generated by the model
ConversationFree-form NegotiationThe agents talk until they agree
SelectLLM-Pick then Sender-StopThe sender's model picks chunks each round and decides when to stop
RequestPeer-Request then Receiver-StopThe receiver says what it lacks, and the holder's model picks
EscalateLLM-Rank then EscalateThe model ranks once, then 1,2,4,8,161, 2, 4, 8, 16 chunks go until both answers stop changing
HandshakeLLM-Rank then Probe-EscalateAnswers and two probe chunks go first, escalation only on disagreement
GuideSize-and-Deadline RoutingChooses a protocol from the context size and the deadline
newBM25-Rank, Dense-Rank or Hybrid-Rank then Top-kkA retriever ranks, and the holder sends its top kk chunks in one round
newBM25-Rank, Dense-Rank, Hybrid-Rank, LLM-Pick or Oracle then Escalate, and LLM-Rank or Oracle then Top-kkFurther pairs that separate the two decisions

New experiments. All use the meter, the pairing and the statistics of the original submission.

  • Five models. The scale axis now covers Qwen3-8B, Llama-3.1-8B, Mistral-7B, Granite-3.3-8B and Phi-4-mini. Context Handover declines with size on all five, losing about half to three quarters of its accuracy by the largest size that fits the window. The accuracy threshold moves with the model.

  • Retrieval rankings. We added BM25, a dense encoder and their hybrid. A retriever scores each chunk separately and never reads the whole context, so it loses far less accuracy with size than Context Handover. With Qwen3-8B on HotpotQA, Dense-Rank then Escalate is the most accurate deployable protocol from about 4K tokens on, and within a few points of the best one at smaller sizes. Its cost is a small encoder and an index on each agent, and an agent without them can rank with its own model.

  • Separating the two decisions. With the schedule fixed, the ranking sets the ceiling. Under an oracle ranking, Escalate at 31.6K tokens is as accurate as Context Handover at the natural size, while the model's own ranking decays with size. With the ranking fixed, the schedule matters when the ranking is imperfect. Under the model's own ranking, Escalate is clearly more accurate than a fixed Top-4 from the natural size on, whereas under Dense-Rank it adds only a few points. A larger fixed amount does not replace the schedule. When a task's need is unknown, a fixed amount collapses once the need exceeds it, while Escalate does not, at the price of extra rounds.

  • Datasets. The scale axis rebuilt on 2WikiMultiHopQA shows the same shapes. On the LoCoMo conversations, which have no paragraphs, LLM-Rank then Escalate keeps the F1 of Context Handover, and the size of the unit decides how much it saves.

  • Threshold. In our tests so far, moving the routing threshold or reusing it on another model changes accuracy little, because the two sides are close in accuracy near the threshold.

Taken together, no protocol is the best everywhere. The paper therefore reports every pair of ranking and schedule, and Size-and-Deadline Routing runs the protocol that answers the most episodes correctly within the deadline (Section 6.5, Figure 6).

Other changes to the writing. Section 3.1 keeps only an explanatory paragraph, and the definitions the main text does not need moved to Appendix B. Section 5 introduces the rankings, the schedules and the design space they span, with a new schematic (Figure 2) and a table of definitions (Table 1 in Appendix D). A new figure in Section 5.3 (Figure 3) walks through one real episode.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.