acceptodds
Under review as a conference paper at ICLR 2027

Lost Between Agents: How Do Task Instructions Drift Across LLM Agents?

Abstract

Multi-agent LLM systems solve complex tasks through communication, and successful collaboration depends on whether important task information is preserved as messages pass between agents. Existing evaluations mainly measure final task performance or overall message similarity, but they do not show what changed during communication or whether those changes actually mattered. This gap raises a key question: how can we identify information loss in agent communication and determine which changes affect downstream behavior? Importantly, a change that occurs frequently is not necessarily the one that matters most to the receiving agent. To address this question, we study sequential task specification handoffs and introduce CommAudit, a protocol for auditing how specifications change as they pass between agents. CommAudit measures both how often task constraints change, which we call exposure, and how much those changes affect the executor, which we call impact, using matched repair and control interventions. Using this protocol, we find that repairing the changes identified by CommAudit consistently improves executor accuracy over stylistic controls across communication agents powered by four different LLMs. Moreover, across three benchmarks and four communication agents, the most frequent change type differs from the one with the largest downstream effect in six of twelve settings, while the same change counts can still lead to different repair outcomes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model Agents, Multi-Agent Communication, Task Specification Relays

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_LmrwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper Summary

The paper studies how task specifications degrade when relayed across LLM agents. Using an 8-hop paraphrase chain with a fixed executor, the authors propose CommAudit. CommAudit labels each atomic constraint as lost, mutated, weakened, or added relative to the original specification. It then measures two quantities:

  • Exposure: how often each type of change occurs.
  • Impact: the executor's accuracy gain when a change type is repaired, compared against a matched stylistic control.

Across three benchmarks and four communication agents, relays degrade accuracy, mostly in the first few hops. Repairing the annotated changes recovers accuracy on BigCodeBench. The most frequent change type differs from the most harmful in 6 of 12 settings. A formal result states that change counts predict impact only if impact is constant among relays with identical counts.

Critical strengths

Critical Strengths

  • The repair-versus-matched-control design is a clean way to separate consequential constraint changes from surface rewording.
  • Supporting checks are reasonable. Repair closes 89% of the accuracy gap between agents, the effect persists when the executor is swapped, and annotation typing shows good agreement (κ=0.849).
Critical weaknesses

Critical Weaknesses

  1. The insights are limited. The headline finding, that change frequency differs from change impact, is close to self-evident. In practice it reduces to "additions are frequent but mostly harmless, while losses are rarer but harmful," which matches intuition. The theory adds little. Proposition 1 restates the data-processing inequality, and Proposition 2 is essentially definitional: counts predict outcomes if and only if equal counts imply equal outcomes.

  2. The analysis offers little actionable guidance. The paper proposes no mitigation, and the design implications in Appendix I are generic. As an analysis paper, it does not show how its findings should change the design of multi-agent communication.

  3. The "6 of 12" result is fragile. Many impact estimates are around ±0.02, and their confidence intervals include zero. Taking the argmax over such noisy, near-zero values is unreliable, and only 4 of the 6 disagreements are robust (p_mis ≥ 0.89). The within-fiber variance test faces a similar issue: it uses a binary outcome over small fibers, so the measured variance largely reflects noise.

  4. The repair intervention may be confounded. The repair model sees the original specification x0, while the control model does not. Repair could therefore restore more than the annotated changes, and repaired accuracy nearly returns to the original baseline A0. The paper does not verify that repairs touch only the listed changes.

  5. The setting is artificial and one model plays several roles.

    • Eight hops of free paraphrasing rarely occur in real multi-agent systems.
    • DeepSeek V4-Flash serves as annotator, repair model, control model, and executor.
    • Change detection itself is not validated.
    • The paper overlaps substantially with prior work on iterated distortion and delegation corruption (Mohamed et al., 2025; Laban et al., 2026).
Questions for authors

Questions For Authors

  1. Beyond "frequency ≠ impact," what non-obvious insight does CommAudit reveal? How should practitioners change their system design based on these results?
  2. Do repaired specifications differ from the hop-8 relay only at the annotated changes? Please report a human or automatic check.
  3. Do the findings hold in a realistic delegation setting, such as a planner handing a task to a coder, rather than repeated free paraphrasing?
Reviewer_9YFQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies how task instructions change as they get rephrased by an LLM multiple times and how this impacts the performance of an executor on that task. The paper then proposes CommAudit to address the loss of critical information or addition of irrelevant task specification and repair the rephrased tasks. The evaluation is performed on 3 benchmarks, with variety of LLMs rephrasing tasks, and one fixed executor.

Critical strengths
  • Mostly the presentation and writing is okay (see weaknesses for some suggested improvements).
  • A good number of models are evaluated for the rephrasing.
  • The variety of the rephrasing prompts is useful.
  • The 3 chosen benchmarks are reasonable.
Critical weaknesses

I am concerned about the practical relevance of this work with 8 hops and unrealistic prompts, it is unclear how much the theory adds, and main results rely on small task differences and confidence intervals that frequently include zero. That said, if my concerns can be addressed I would be happy to raise my score.

Main concerns:

  • While mostly easy to follow, the mathematical results and propositions seem to overcomplicated things and introduce a lot of maybe not so necessary annotation which makes it harder to parse those parts. It is also unclear how important the propositions and theory are for the experiments and empirical observations of the paper. Could the authors please explain how these propositions directly impacted the experiments and were empirically tested / observed?
  • I think a better discussion of the practical motivation is needed. Clearly, the study of multi-agent systems is important, but in practice 8 hops seem highly unlikely, e.g. an orchestrator would use one subagent or maybe that subagent spawns another sub subagent. Additionally, the prompts used here for rephrasing seem quite unrealistic, e.g. something along the lines of "Delegate this task to a subagent" might be more realistic.
  • The same model is used as the annotator, repairer, control generator, and executor. Reporting results for at least one additional executor and different annotator and repair model family would show that the results are more general and not due to a single model.
  • It is also unclear to me how this method would be deployed in practice to make multi-agent systems more robust as it needs x_0, as in it allows auditing of the rephrasing offline while critical part to me seems to be catching these online. Could the authors please comment on this?
  • It would be good to describe the repair and control in the main text as those form a core component of the paper.
  • What is the point of table 1 and how does it related to the core contributions of the paper? If I understand correctly the aim there is to identify the different LLMs?
  • I am also concerned about the statistical significance of the results. In table 4 for most of the CIs for population impact include 0 and core results seem to differ only by a small number of tasks. Running multiple samples per task would also help with establishing statistical significance.
  • It would be interesting to know the proportion of impact that is due to name or interface changes on BigCodeBench, as those would just fail immediately.

Other:

  • The paper frequently uses hard to understand terminology such as "source-anchored" and "executor-relative".
  • Could the authors discuss the addressed gap relative to prior iterated-relay work in more detail.
Questions for authors

See above.

4 comments

Dear Reviewer 9YFQ,

Thank you again for your thoughtful and constructive feedback. As the rebuttal period will end soon, we would like to briefly follow up on our responses to your questions. If any points remain unclear or require further clarification, we would be happy to address them before the discussion period ends.

Best regards,

The Authors

Reviewer_9YFQOpenReview ↗

Response to Authors

Thanks to the authors for the rebuttal. The additional experiments are appreciated though I am afraid a significant portion of my concerns (especially some critical ones) remains unaddressed. These include but are not limited to the unrealistic prompts (delegation was not tested), statistical significance of the per-change-type results, models apparently not being able to follow the verbatim instruction (except on the one quoted benchmark), thin experimental support for the theory, small number of tasks for additional experiments (editor swap), A4 proposed approach seems more realistic but is untested.

Thank you for taking a close look at the additional experiments and for clarifying the concerns that remain.

1. Delegation and the online check in A4. Every system that delegates through natural language relies on a fundamental communication step. Whether a planner assigns a whole task or a single subtask, the receiving agent needs the requirements that govern its part of the work. A good planner may compress the brief considerably. Our controlled setup lets us ask which details it can safely compress. In one BigCodeBench example, the original specification named the input keys to check and required a ValueError if either was missing. The handoff simply asked for “a meaningful error.” Code generated from that brief failed the tests. Restoring the lost requirements yielded passing code, while a matched stylistic edit still yielded a failure (Appendix J). This gives A4 a concrete motivation: a planner could retain the freedom to rephrase while checking that the outgoing brief preserves requirements such as the error contract. Our experiments measure the value of restoring those requirements. Your proposed delegation prompt and an online evaluation would extend this to how often a planner omits them and how reliably a check catches them before dispatch.

2. Statistical significance by change type. The two rows in A7 answer different questions. Across all Qwen IFEval tasks, the confidence interval leaves the average advantage of loss repair uncertain. For a system deciding what to fix in a particular handoff, however, the choice between restoring an omission and removing an addition arises only when both are present. On those tasks, the paired comparison favors restoring the omitted requirements, with a confidence interval that excludes zero. This gives a concrete setting in which the evidence supports prioritizing one repair, while leaving the broader ordering open.

3. How the theory informs the experiments. Proposition 1 provides an ideal-receiver benchmark for interpreting what information can survive a relay. Proposition 2 has a direct consequence for the analysis: if change counts are sufficient to predict expected repair impact across tasks, handoffs with identical counts must agree in that expected impact. This motivated us to compare outcomes within groups matched on all four counts (§5.4 and Appendix G.4). Appendix J illustrates why that comparison matters. Two handoffs each lost exactly two requirements, yet their observed repair outcomes went in opposite directions. A system deciding what to check would see the same counts in both cases. Measuring what happens after repair gives it information the counts alone leave out.

4. Following the verbatim instruction. Table 3 shows that under the same copy instruction, downstream accuracy stayed close to the original-task baseline on BigCodeBench and fell substantially on IFEval. This matters in practice because asking an agent to forward a task verbatim can still leave downstream performance sensitive to the benchmark. Establishing how often the sender reproduced the source word for word would require comparing the handoff text directly with the source.

5. What the editor swap establishes. We used the same 25 BigCodeBench tasks for both editors, holding the handoffs, annotations, and executor fixed. This paired design lets us ask whether the repair gain on those tasks depends on which model writes the edits. With either editor, repair outperformed the matched style control for all four rewriters.

Reviewer_9YFQOpenReview ↗

Response to authors

Thanks for the follow-up. My stated concerns still remain and would need additional experiments to address them, which were not provided. That said, I appreciate the author's efforts and will make my final decision during the AC reviewer discussion.

Reviewer_WcdNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CommAudit, a framework for analyzing information loss in multi-agent communication and its impact on downstream task execution. It tracks changes in atomic task constraints, including loss, mutation, weakening, and addition, and evaluates communication quality from two perspectives: Exposure, which measures the frequency of constraint changes, and Impact, which measures their effects on downstream performance through matched repair and control interventions.

Critical strengths
  1. The paper is generally well organized, with a clear presentation of the motivation and methodology.

  2. The distinction between exposure and impact provides a useful perspective for evaluating instruction drift in multi-agent communication.

Critical weaknesses
  1. The paper primarily focuses on developing an evaluation framework for analyzing instruction drift, while offering relatively limited insights into its underlying mechanisms. Although the authors identify a discrepancy between exposure and impact, further analysis of why certain constraints are more consequential and why different agents exhibit distinct drift patterns would strengthen the paper's contributions.

  2. The experiments mainly consider linear communication chains in which LLMs repeatedly paraphrase task instructions. This setting is relatively simplified compared to practical multi-agent systems involving heterogeneous roles, task decomposition, and dynamic interactions. Additional experiments in more realistic MAS scenarios would help establish the generalizability of the findings.

  3. CommAudit uses LLM-generated annotations to identify and classify constraint changes. However, the reported agreement analysis mainly evaluates change-type classification, without independently assessing the precision and recall of change detection. Further validation of the annotation process would help improve confidence in the exposure and impact measurements.

  4. The Repair and Style Control interventions are primarily matched by the number of edits, while other factors such as text length, structure, and surface properties are not explicitly controlled to the same extent. A more detailed analysis of these factors would help strengthen the interpretation of the intervention results.

  5. The theoretical analysis builds on established Blackwell information theory and introduces a distribution-independent condition for exposure to fully represent impact. However, this condition is relatively restrictive, and its violation does not necessarily imply that exposure cannot effectively predict impact under specific task distributions. Further discussion of the practical implications of these theoretical results would be valuable.

  6. The paper represents task instructions as relatively independent atomic constraints, while practical tasks may involve dependencies among different constraints. Repairing one type of change in isolation may not fully restore the original task semantics. A more systematic analysis of interactions among constraint changes would provide a more comprehensive understanding of their downstream effects.

Questions for authors

See Weaknesses.

1 comment

Dear Reviewer WcdN,

Thank you again for your thoughtful and constructive feedback. As the rebuttal period will end soon, we would like to briefly follow up on our responses to your questions. If any points remain unclear or require further clarification, we would be happy to address them before the discussion period ends.

Best regards,

The Authors

Reviewer_ov2jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper conducts a set experiments aiming to show how task specifications change and effects of those changes between multiple LLM agent. They attribute the problem into two dimensions: exposure and impact to capture various types of changes frequency and corresponding effects. The experiemnts are conducted on three benchmark datasets on four LLMs. The results shows most of the information loss happens in the first two or three hops. However, limited experiement on various LLMs and configurations limit the contribution of this emprical or evaluation paper.

Critical strengths
  • The focus research question is interesting and meaningful. This paper introduce new dimension to check what kind of information changed and whether those changes are important or not.
  • The defined two dimensions are insightful. To seperate the changes and effects can give more resuable direction for future research.
  • The main cliams can be supported by the experiments (ingoring the limited model families, only on the reported). The experimental design is good.
Critical weaknesses
  • The evaluation setup is not comprehensive. The paper only evaluated on four communication models, most of are light and small models. Different size, model types (reasoning, instruct-tuned only, moe, etc) should be considered, especially considering local deplyment.
  • In addition, the executor, annotator and repair model are using the same pipeline. This limits the contribution and generalisability.
  • The sample size and task scope is limited. More complicated, cross-domain, diverse scenraiso multi-agent system evaluation are expected.

In summary, the paper proposed a good view to analyse or evaluate multi-agent system information changes and effects. But the conducted experiements and evlauation setup, not that comprehensive to support their contribution.

Questions for authors
  • Does CommAudit work as well in complex settings, in other domains, and in other languages?
  • How do results differ across model families and sizes?
1 comment

Dear Reviewer ov2j,

Thank you again for your thoughtful and constructive feedback. As the rebuttal period will end soon, we would like to briefly follow up on our responses to your questions. If any points remain unclear or require further clarification, we would be happy to address them before the discussion period ends.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.