acceptodds
Under review as a conference paper at ICLR 2027

Lost in Communication: Uncertainty Propagation in Multi-Agent Systems

Abstract

In multi-agent systems of large language models, the agent that observes uncertain evidence is often not the one that acts on it, so the uncertainty must propagate through hand-offs. We propose a controlled experimental design for uncertainty propagation, implemented in UPMAS. Only the irreducible uncertainty of the evidence varies, and systematic differences in behavior are causally attributable to this uncertainty. Across three open-weight models, uncertainty metrics barely react to the induced uncertainty, even at the agent that observes the evidence, and what little they register fades within a few steps. In their messages, agents omit the information three times as often after the first hand-off as before (37% against 11%) and otherwise mostly assert one possible state of the world as certain. We call this loss of uncertainty between the evidence and the actions that depend on it vanishing uncertainty. The acting agent’s own uncertainty quantification cannot recover uncertainty lost upstream, so multi-agent systems must propagate uncertainty through their messages.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Uncertainty, LLM agents, Multi-agent, Safety, LLM

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UAmjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors study whether uncertainty in external evidence survives communication between agents in MAS. The authors introduce UPMAS, a controlled enterprise workflow with various families of uncertainty manipulations, and evaluate three open-weight models using numerical uncertainty measures and linguistic analysis. They report “vanishing uncertainty” phenomenon, which refers to downstream messages omit contingencies or present one possible outcome as certain.

Critical strengths
  1. Important and timely problem. Preserving uncertainty across agent hand-offs is a significant challenge for practical multi-agent deployment.

  2. Useful empirical framework. The deterministic tools, controlled changes to external evidence, and explicit tracing of information through downstream quantities provide a thoughtful setting for investigating this problem. The distinction between preserving a claim and preserving its uncertainty is particularly useful. The message-based evaluation could also be applied to black-box systems, although the reported experiments use open-weight models.

  3. Informative analysis of communicated content. RQ3 directly examines whether messages retain plausible alternatives and relevant contingencies. This is the strongest part of the paper and provides evidence of a practically meaningful failure within the evaluated workflows.

Critical weaknesses
  1. Mismatch between the intended uncertainty and its measurement. A model can recognize that the delivery date is uncertain and still be confident that its answer is correct. For example, “Delivery could be Wednesday or Saturday, depending on stock” correctly expresses uncertainty. Therefore, high confidence or consistent answers do not necessarily mean that the model failed to recognize uncertainty. The authors should verify that their metrics capture uncertainty about the external information, rather than confidence in the answer itself.

  2. The uncertainty levels need stronger justification. In Fig. 2, lower stock does not necessarily mean greater uncertainty. For example, a 99% chance of stockout is more certain than a 50% chance, even though the risk is higher. The authors use expected consumption to explain why their high-uncertainty condition leaves both outcomes plausible, but this depends on assumptions about how much stock has been consumed since the last update. These assumptions need stronger validation. Without it, the experiments show how agents respond to changes in the evidence, but do not clearly isolate the effect of uncertainty itself.

  3. Limited comparison with general information loss. RQ3 provides useful evidence that uncertainty-related qualifications disappear during communication. However, the paper does not compare their retention with that of other relevant information. Such a comparison would help determine whether these qualifications are especially vulnerable to loss or whether the observed behavior reflects a broader tendency to omit details during agent hand-offs.

Questions for authors

The problem is important, and the message-level findings are promising. However, the measurements and their interpretation do not clearly distinguish environmental uncertainty, confidence in an action, and information loss during communication. I would be inclined to raise my score if the authors adequately address or clarify these concerns.

4 comments

We thank Reviewer UAmj for taking the time to read our manuscript and providing constructive feedback. We are glad they recognize that the problem is important, that our framework is useful and its message-based evaluation also applies to black-box systems, and that our analysis of the messages shows a practically meaningful failure.

W1: The metrics may measure confidence in the answer rather than uncertainty about the external information.

We thank the reviewer for this important distinction. We now make it explicit in the main text, and a new ablation shows that it does not change our results on the uncertainty metrics. Section 2 (L188-L196) now defines the three constructs the reviewer distinguishes, uncertainty in the environment, confidence in an action, and loss in communication, together with their measures. The numeric metrics measure the confidence of an action, so a hedged action such as the reviewer's example can be stated confidently. For this reason, RQ3 reads the content of the actions: the Information Tracking Judge scores which outcomes an action still admits about the external fact (coverage) separately from how assertively it states them (concept assertiveness). The two diverge in our results. At medium and high, coverage falls to about 0.45 within five carrier actions, while concept assertiveness stays near a bare assertion at every level (S3.3, L445-L456). This matches prior work, in which models tend to not hedge even when uncertain (Yona et al., 2024; Zhou et al., 2024). To verify that the RQ1 result does not stem from confident hedged actions, we repeat it on the actions that commit to one outcome, for which the risk rises. The result holds. On these actions, the metrics respond to the level as weakly as on all actions of the origin agent, and every 95% interval of the difference includes zero (new App. F.1.4, L3607-L3617). The agents also recognize the uncertainty. No first action after the evidence omits it (S3.3, L472-L473), and when asked directly, each agent model rates the facts as more uncertain at high than at low in all nine families (App. D.9, L2453-L2457). The loss therefore lies mainly in carrying the uncertainty, not in recognizing it.

W2: The uncertainty levels need stronger justification.

We now justify the levels in the main text and in a new per-family appendix, and an independent check with three LLMs supports their ordering. We thank the reviewer for the 99% example, which pinpoints what the levels order. They order how evenly the evidence splits the two possible outcomes (branches), not how far the evidence shifts from low. Low clearly favors one branch, and high leaves both branches plausible, so a 99% chance of stockout would count as less uncertain than our high level. We now state this in the main text with Figure 2 (S3.1, L230-L236), and the new appendix argues for each family why its levels order the uncertainty (App. C.2.3, L1793-L1910). In the reviewer's example, the reported stock leaves less room for consumption at each level, so stockout grows more likely under any consumption rate. The agents can estimate the consumption rate from the evidence itself, because the tool response that reports the stock also gives the count and time of the last refill (App. C.2.3). To validate these arguments, three frontier LLMs that do not serve as agents, blind to the level, estimated the probability of each branch for every task from the origin evidence and the records needed to read it (App. C.3, L1912-L1967). The judges form their own estimates, so the check does not rest on our consumption assumption. Each of them rates high as more uncertain than low in all nine families, and their mean uncertainty rises from low to medium in all nine.

W3: Limited comparison with general information loss [...]

We thank the reviewer for this suggestion and added the comparison, which shows that agents pass on a fact but drop its uncertainty. As a first comparison, we use information we already track, the fact that the uncertainty is about. This fact has the same relevance, the same path, and the same carriers as its uncertainty. After the first hand-off, at medium and high, 85% of the required carriers state the target concept, but only 2.5% keep both of its branches open (new App. F.3.4, L3953-L3971). This matches human communication, in which the claims survive but their hedges are lost (Fahnestock, 1986; Graeber et al., 2025). The uncertainty can only travel with its fact, so the fact caps how often the uncertainty is kept. At 85%, this cap lies far above the 2.5% we observe. The comparison therefore suggests that agents omit the uncertainty of a fact at higher rates than the fact itself.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Thank you to the authors for their response. For W3, I think it is better to compare the uncertainty with other secondary details unrelated to it (e.g., quantities, identifiers, or constraints), rather than with the fact it qualifies. The core fact is expected to survive a hand-off more often than any qualifier attached to it, so the gap between 85% and 2.5% does not show that uncertainty is more vulnerable than other details.

The response addresses my concerns on W1 and W2, so I have raised my score from 2 to 3.

We thank the reviewer for raising their score and for the suggested comparison, which we have now run.

W3: Agents keep every other detail of the case at least 5.7x as often as the uncertainty. We consider the same actions as before. On these actions, we check whether the agent restates details of the case that carry no uncertainty and that it has in context: four identifiers, a date, and three quantities.

Kept after the first hand-offKindShare
Serial numberIdentifier62%
Purchase dateDate60%
Customer emailIdentifier38%
Product priceQuantity27%
Package weightQuantity25%
Registration IDIdentifier16%
Stock countQuantity15%
Customer IDIdentifier14%
Uncertainty2.5%

We added the table to App. F.3.4 (Table 17, L4050-L4068). While these facts are lost more frequently than the target concept with 85%, this evidence still suggests that secondary details unrelated to the uncertainty are lost more frequently.

Presentation: We have identified concrete issues and will fix them in the final version. Among others:

  1. Repeated and split explanations in Sections 2 and 3. What the levels order is explained twice, in S2 ("the levels order the risk by construction", L185) and in S3.1 ("how evenly the evidence splits [the branches]", L230-L231). The numerical and linguistic measurements are introduced in four places (L120-L122, L188-L196, L311-L312, L320-L323). We will explain each once.
  2. Terminology. Some terms are defined but not used later, such as the uncertainty path (L168-L169). We will remove such unused definitions and provide figures to visualize the remaining.
  3. Results. We will rewrite jargon-heavy sentences, such as "Slightly negative values are estimator bias rather than anti-persistence" (L407-L408), and define the judge measures of RQ3 before the example that uses them (L432-L437, defined only at L446-L459).
Reviewer_p7yKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies multi-agent LLM systems, where the agent that sees uncertain evidence is often not the agent that acts on it. The acting agent then learns about the uncertainty only from messages from other agents. The authors argue that current uncertainty methods cannot show this loss, because they measure a model’s confidence in its own output.

The authors’ main research question is whether a multi-agent system keeps uncertainty from where it enters to where an agent acts on it. They split it into three sub-questions: whether uncertainty metrics react at the agent that sees the evidence, whether they pass the uncertainty across agents, and whether agents carry it in the content of their actions. The paper gives no formal hypotheses, but it does expect, for example, that uncertainty measures do not go down when the level of uncertainty rises.

In my reading, the main contribution is the controlled design, which is useful. The empirical contribution is more limited. The results come from one scenario with nine uncertainty families and three agent models, and the uncertainty metrics give mostly null results. The step beyond earlier work is modest, and the explanation that hand-offs cause the loss is not tested in isolation.

The validation uses the controlled design, so differences between uncertainty levels can be linked to the uncertainty itself. The uncertainty metrics barely react, and after a few steps the messages usually state only one outcome. The message results are the stronger evidence. However, the narrow setup limits their support for the authors’ conclusion that the loss is specific to multi-agent systems..

Critical strengths
  1. Controlled experimental design. The authors vary uncertainty that no tool or agent can resolve, at three levels, and keep everything else the same. This makes it possible to tell uncertainty that was lost from uncertainty that was resolved or never seen, which normal trace analysis cannot do. This matters for my assessment because it is the main and most reusable part of the paper.

  2. Measuring the loss in the content of actions. An LLM judge checks whether actions still cover both possible outcomes, how confident they sound, and whether they leave the information out. This shows a loss that standard metrics miss. It is the paper’s main evidence, but it has the limits I describe under the weaknesses.

Critical weaknesses
  1. The central claim is shown only indirectly (soundness).

The paper defines lost uncertainty as uncertainty missing at the final action, but the main text does not report final actions separately. The evidence comes from positions along the path, and later positions hold only longer trials and only actions that still state the concept.

The filtering is also uneven: trials are removed only at medium and high (7–8%, plus the clarification trials), none at low, and there is no sensitivity analysis.

This matters because the paper’s own endpoint is not measured directly, so it is unclear how much uncertainty is lost at the point of action.

  1. The explanation is untested and the novelty is modest (originality, significance).

The paper’s main explanation is that hand-offs cause the loss, but it does not test this in isolation: the same tasks are not run with a single agent. The planned comparison between the agent that sees the evidence and the agents after the first hand-off fits a hand-off effect, but it cannot fully separate hand-offs from distance to the evidence or from longer generation.

Work cited by the paper already shows that information changes when LLMs pass it along a chain (e.g., Mohamed et al., 2025; Perez et al., 2025). The paper also does not discuss uncertainty absorption (March and Simon, 1958), a closely related idea from research on organizations. The results come from one scenario, nine uncertainty families and three agent models, so they do not yet support a general result, and the main new contribution is the controlled design.

This matters because, without a tested explanation, the paper does not yet show why the loss happens or how to prevent it.

Questions for authors
  1. Can you report results at the final actions separately, by model and level, including the orchestrator’s section on unresolved uncertainties?

  2. How do the results change if the trials removed by the output checks at medium and high, and the clarification trials, are kept in the analysis?

  3. What does the paper show about uncertainty that the cited work on information change along chains of LLMs does not?

  4. Can the existing data separate hand-offs from distance to the evidence, for example by comparing actions at the same position with and without a hand-off before them?

4 comments
AuthorsOpenReview ↗

Rebuttal 1/2

We thank Reviewer p7yK for taking the time to read our manuscript and providing constructive feedback. We are glad they recognize that our controlled design supports causal claims about uncertainty and that measuring the content of actions shows a loss that standard metrics miss.

The paper gives no formal hypotheses [...]

We thank the reviewer for this comment and now state one central hypothesis before the research questions, operationalize it more clearly with one testable prediction per research question (S2, L199-L202).

W1: The central claim is shown only indirectly (soundness).

W1a & Q1: The paper's endpoint, the final action, is not reported separately.

We now report the final actions separately, by model and level, and they confirm the loss at the point of action. We agree that the paper's own endpoint should be reported directly, not only through the position curves. No UQ metric rises with the level at the final actions for any model (Figure 20). In their content, the final actions at medium and high commit to one branch about as often as at low (75 to 100%) and rarely keep both branches open (at most about 10%; new Figure 29). The final report, scored as a whole including the orchestrator's section on unresolved uncertainties, keeps both branches open in none of these trials. Unlike the position curves, these readouts cover every trial, whatever its length, and Figure 29 counts omissions as a separate outcome.

W1b & Q2: Trials are removed only at medium and high, with no sensitivity analysis.

We added the requested sensitivity analysis, and no conclusion changes (App. D.5, L2254-L2291). We agree that removing trials only at medium and high calls for this check, and repeat both final-action analyses with the removed trials put back in, including the clarification trials. In content (Table 8), the share of final actions that commit to one branch, keep both branches open, or omit the concept moves by at most 4 percentage points (pp) and by less than 0.01pp in 73% of the cells that the restored trials enter. In the UQ metrics (Table 9), the origin-only change from low stays within the 95% interval of the strict analysis in all but one cell. Of the removed trials, only the clarification trials and one escalation handled the uncertainty faithfully (S3.3, L483-L487). Even if all of them had kept both branches open, they would raise the share of medium and high trials that do so by at most 3 percentage points. The origin check cannot bias the comparison between levels, because the agent makes the origin call before it sees the manipulated response.

W2: The explanation is untested and the novelty is modest W2a & Q4: The hand-off explanation is not tested in isolation.

We now state the hand-off explanation as a hypothesis (S4, L501-L506) and added an analysis that finds no support for one of its two alternatives, longer generation. The comparison of the origin agent with the agents after the first hand-off indeed cannot separate the hand-off as source of the loss from distance to the evidence or from longer generation. We appreciate the comparison suggested in Q4, but the structure of our task leaves too little data for it. In our system, the actions that must carry the target concept are almost only the hand-off messages and the final actions, so the position on the carrier path and the number of hand-offs coincide. Required carriers without a hand-off before them occur only at the start of the carrier path, so the two groups overlap too little for a test. The data can, however, test longer generation. The orchestrator keeps its context for the whole trial, while a specialist starts every delegation with a fresh one. If long contexts caused the omissions, the orchestrator should omit the concept more often. After the first hand-off, both omit it about equally often (orchestrator 13.0%, specialists 15.4%, +2.5 pp, 95% CI [−3.4, +8.5], App. F.3.2, L3933-L3940). The omission rate also falls with the number of actions and tokens an agent generated before the omitting action, but survivor bias and a changing mix of families shape this trend, so we do not read it as evidence.

AuthorsOpenReview ↗

Rebuttal 2/2

W2b & Q3: Work cited by the paper already shows that information changes when LLMs pass it along a chain [...]

We now discuss information loss along chains of LLMs and in human and organizational communication (App. A, L1017-L1056; S1, L65-L67), and our work goes beyond it in three ways. We appreciate this question, as the comparison sharpens what our work adds. First, we measure the uncertainty about a fact rather than the fact itself (in our case the target concept), and the two come apart. After the first hand-off, at medium and high, 85% of the required carriers state the target concept, but only 2.5% keep both of its branches open (App. F.3.4, L3953-L3971). Second, we induce irreducible uncertainty at a known origin of a tool-using MAS and hold everything else fixed, so any uncertainty missing downstream was lost inside the system. Chain work relays given text and observes how it drifts. Third, we show what losing the uncertainty between agents means for UQ. An agent's UQ can only reflect the uncertainty in its context, and in a MAS other agents write that context. Even at the origin, the metrics we test barely register the uncertainty (RQ1), and little of what they register crosses hand-offs (RQ2). A better UQ method may help at the origin, but not downstream. Once the messages commit to one branch of the evidence and its inferences, a later agent's confident action fits what it received, but not the evidence. Measuring the loss of uncertainty and qualifying statements rather than facts, and linking it to UQ, sets our work apart from research on information loss.

We thank the reviewer for pointing us to uncertainty absorption (March and Simon, 1958), an important concept from social and organizational sciences. It gives a theoretical foundation to how we trace uncertainty, and we now build it into our definitions (S2, L162-L175) and discuss it in the related work (App. A, L1032-L1056). Absorption occurs when an inference replaces the evidence and is passed on as fact. Our lineage tracks these inferences, and a faithful propagation would let an inference replace the evidence only if it keeps both branches open.

W2c: The results come from one scenario, nine uncertainty families and three agent models, so they do not yet support a general result, and the main new contribution is the controlled design.

We now state the scope of a single scenario in the main text (S4, L506-L508), and two further experiments add breadth. We agree that a single scenario shows that the failure mode exists under controlled conditions, not how general or frequent it is. Within the scenario, the nine families vary where the uncertainty enters, which of four agents first sees it, and its form (ambiguous, stale, conflicting, or hedged), so the evidence spans several paths of diverse information through the system. In a two-agent system on five Q&A datasets, the single hand-off experiment (App. B.2.1, L1217-L1221) finds that the likelihood metrics barely cross a hand-off. We are also running a mixed-model experiment and will report the results in this thread.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

AuthorsOpenReview ↗

Addendum - Heterogeneous Systems

Mixed-model experiment (W1). We completed the mixed-model experiment, and with different models in the agent roles (orchestrator vs. specialist), the uncertainty metrics again do not reflect the uncertainty at the final actions (new App. F.1.5, L3618-L3638). The information changes models at every hand-off, many times per trial, because the specialists communicate only through the orchestrator. We rotate the three models through the two roles, so that each model runs the orchestrator once and the specialists once, and every hand-off crosses a model boundary. At the final actions, where the agents act on the uncertainty, we find no systematic trend with the uncertainty level in any metric or configuration (new Figure 24) and results closely resemble those we already report for the single-model experiments. Propagation through the uncertainty metrics thus fares no better in heterogeneous systems.

Reviewer_Z1WqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether multi-agent LLM systems carry uncertainty from the agent that observes ambiguous evidence to the agent that acts on it, and finds that they largely do not. Using a controlled design called UPMAS that varies only the irreducible uncertainty of one piece of evidence while holding everything else fixed, the authors show across three open-weight models that uncertainty quantification metrics barely respond to induced uncertainty and that whatever signal appears does not persist across hand-offs. In messages, agents omit the relevant information about three times as often after the first hand-off as before, and within roughly five actions they typically assert a single possible outcome as confidently as if the evidence were certain; the authors call this failure "vanishing uncertainty." Because the acting agent's own uncertainty quantification cannot recover information lost upstream and each hand-off resets context, they argue that multi-agent systems must propagate uncertainty explicitly through their messages, pointing to language as the key channel to work on while acknowledging open questions around cost and privacy .

Critical strengths

UPMAS varies only the irreducible uncertainty of one piece of evidence and holds everything else fixed, so differences in behavior can be causally attributed to the uncertainty itself rather than guessed from observed traces.

It treats uncertainty in multi-agent systems as a propagation problem and names its failure mode, "vanishing uncertainty," giving the field a clear target.

It reads uncertainty both numerically (UQ metrics) and linguistically (confidence of wording, completeness of information), connecting UQ to the literature on how language communicates uncertainty.

Critical weaknesses

The study uses one model in every role; in heterogeneous systems the metrics are not even comparable, and nothing suggests propagation fares better there.

How to actually make uncertainty propagate remains unanswered, the paper diagnoses the failure rather than solving it.

It studies uncertainty from external sources (task and environment), not from the model's parametric knowledge, in settings where agents cannot check facts against their own knowledge.

Because missing uncertainty in a raw trace is ambiguous (lost, resolved, or never seen), the method needs a purpose-built, controlled harness rather than working directly on natural traces.

Questions for authors

see above

2 comments

We thank Reviewer Z1Wq for taking the time to read our manuscript and providing constructive feedback. We are glad they recognize that our controlled design supports causal claims about uncertainty, that treating uncertainty as a propagation problem gives the field a clear target, and that we read uncertainty both numerically and linguistically.

W1: Homogenous system: Single-model across agents

We show that even in the simple setting of a single model, uncertainty propagation already fails. We are running a mixed-model experiment to cover the more complex case of heterogeneous systems, whose results we will report in this thread. We agree that heterogeneous systems matter in practice and that token-level metrics (NLL, entropy) are not comparable across models, which leads us to think that uncertainty would not propagate either. The conclusions of the linguistic analysis (RQ3) are likely unaffected, because it scores the content of actions rather than token probabilities. Our new experiment will cover this setting.

W2: How to actually make uncertainty propagate remains unanswered, the paper diagnoses the failure rather than solving it.

The paper leaves uncertainty propagation as an open problem, because it shows that simple solutions fail, but gives directions for future work. First, passing UQ scores along would carry little, because the metrics we test barely register the uncertainty (RQ1). Second, asking only for a report of unresolved uncertainties does not suffice: the orchestrator is told from the start to include them in its final report, yet no final report at medium or high keeps both possible outcomes open properly (App. F.3.3). Third, the agents largely perceive the uncertainty but do not state it in a natural workflow (App. D.9). Hence, while better perception may help in places, a remedy should mainly target what the agents state. Our current results show what a remedy should target: interventions need to make agents communicate the uncertainty actively and act on it.

W3: External uncertainty only.

We added an experiment indicating that the weak transmission across hand-offs is not specific to external uncertainty (App. B.2.2, L1287-L1321). The main experiment indeed covers external uncertainty, which matters in practice, because deployed agents act on evidence from tools and records. It also enables the causal attribution the reviewer highlights: we can set the uncertainty of external evidence and hold everything else fixed, which we cannot do for uncertainty in the model's weights. Our single hand-off experiment (App. B.2.1, Table 2) covers parametric uncertainty without this control, and NLL and ensemble NLL still correlate across the hand-off at only 0.140.14 to 0.260.26. We find no significant difference between the datasets with parametric and those with external uncertainty in how the likelihood metrics correlate across the hand-off (NLL: Δ=0.03\Delta = 0.03, 0.200.20 vs. 0.230.23), so the weak transmission is not specific to external uncertainty.

W4: Method requires purpose-built data.

We thank the reviewer for raising this and added a guide for applying our design to other MAS, starting from a pre-existing trace (App. C.5, L1984-L2064). Extending to other MAS requires annotated data and active runs, to measure the causal attribution that the reviewer highlights. The guide starts from a pre-existing trace and a replayable environment and covers the manipulations, annotations, and checks. It follows our own approach closely, except that we built UPMAS from scratch. Our other techniques transfer directly: for instance, the confidence judge and the analysis in Appendix B need no annotations or matching.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all three reviewers for their efforts to read our work and constructive feedback, which we have addressed in an updated manuscript (all changes in blue). The revisions have made the paper clearer and its evidence stronger.

Our paper shows that in multi-agent systems of LLMs, the uncertainty of evidence is lost as it passes between agents, and that uncertainty metrics barely react to it. All three reviewers recognize our controlled design, which varies only the uncertainty of one piece of evidence, so that differences in behavior can be attributed to it. UAmj, who replied to our rebuttal, raised their score from 2 to 3. We addressed the reviewers' feedback with numerous new experiments and analyses and a substantial revision of the text. The new results reproduce the failure we report and thus confirm our conclusions.

1. Do the measures and uncertainty levels capture the uncertainty? (UAmj W1, W2; p7yK)

  • UAmj asked us to justify that our manipulations induce graded uncertainty (low, medium, high). We now justify it in the main text and for each of the nine types of uncertainty (S3.1, new App. C.2.3). An independent check confirms it. Three frontier LLMs rate our high-uncertainty conditions as more uncertain than the low ones across the entire dataset (new App. C.3).
  • UAmj noted that an agent can be confident in an answer that correctly states the uncertainty, so high confidence need not mean the agent missed it. This is why we measure the loss in three ways: with numerical metrics, with the confidence the agents state, and in the content of their messages. All three show that the uncertainty is lost. We now separate confidence from uncertainty more clearly (S2). A new ablation confirms that our results hold when we explicitly account for this distinction (new App. F.1.4).
  • p7yK noted that the paper states no formal hypothesis. We now state one, with a testable prediction for each research question (S2).

2. Is the uncertainty lost where the system acts? (p7yK W1, Q1, Q2) Yes. Following the reviewer's suggestion, we report the loss separately at the system's final actions, the last decisions that depend on the uncertainty. This confirms the loss where it affects the outcome, consistent with our main results (new App. F.3.3). The reviewer also asked whether removing trials that never complete a final action biases the results. Including these trials changes no conclusion, so our results are robust to this choice (new App. D.5).

3. Is this more than general information loss? (p7yK W2b, Q3; UAmj W3) Yes. p7yK pointed to prior work on information loss in chains of LLMs and in organizations, which we now discuss to position our contribution (S1, S2, App. A). We detail how our controlled experimental design and our focus on uncertainty in LLM agents go beyond this literature. UAmj suggested comparing the loss of uncertainty with general loss of information. A new comparison shows that uncertainty is particularly vulnerable. Every other detail we track is kept at least 5.7x as often (new App. F.3.4).

4. What causes the loss, and how can it be fixed? (p7yK W2a; Z1Wq W2) p7yK noted that we attribute the loss to the messages between agents without testing alternatives. We now state this as a hypothesis (S4) and find no support for one alternative, long contexts (App. F.3.2). Z1Wq noted that we diagnose the failure without fixing it. We now discuss how our results narrow the search for a fix (S4). Passing on uncertainty scores would carry little, and priming the agents to report unresolved uncertainty did not suffice. Yet every agent model, asked directly, rates the evidence as more uncertain at high than at low (new App. D.9). A fix should therefore close the gap between what agents perceive and what they state.

5. Do the results generalize? (Z1Wq W1, W3, W4; p7yK W2c) Yes, our findings hold in two new settings.

  • Z1Wq noted that one model fills every role. With different models in different roles, the metrics again barely react to the uncertainty at the final actions (new App. F.1.5).
  • Z1Wq noted that we study only uncertainty in the environment, and p7yK that a single scenario limits generality. A new two-agent experiment on five Q&A datasets confirms our results beyond our scenario, for uncertainty in the model's own knowledge as well as in the environment (new App. B).
  • Z1Wq noted that our experimental protocol requires a purpose-built system rather than existing traces. A new guide shows how to apply the protocol to other multi-agent systems, starting from an existing trace (new App. C.5).

We believe these changes address every weakness raised in the reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.