- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Authors study whether uncertainty in external evidence survives communication between agents in MAS. The authors introduce UPMAS, a controlled enterprise workflow with various families of uncertainty manipulations, and evaluate three open-weight models using numerical uncertainty measures and linguistic analysis. They report “vanishing uncertainty” phenomenon, which refers to downstream messages omit contingencies or present one possible outcome as certain.
Critical strengths
-
Important and timely problem. Preserving uncertainty across agent hand-offs is a significant challenge for practical multi-agent deployment.
-
Useful empirical framework. The deterministic tools, controlled changes to external evidence, and explicit tracing of information through downstream quantities provide a thoughtful setting for investigating this problem. The distinction between preserving a claim and preserving its uncertainty is particularly useful. The message-based evaluation could also be applied to black-box systems, although the reported experiments use open-weight models.
-
Informative analysis of communicated content. RQ3 directly examines whether messages retain plausible alternatives and relevant contingencies. This is the strongest part of the paper and provides evidence of a practically meaningful failure within the evaluated workflows.
Critical weaknesses
-
Mismatch between the intended uncertainty and its measurement. A model can recognize that the delivery date is uncertain and still be confident that its answer is correct. For example, “Delivery could be Wednesday or Saturday, depending on stock” correctly expresses uncertainty. Therefore, high confidence or consistent answers do not necessarily mean that the model failed to recognize uncertainty. The authors should verify that their metrics capture uncertainty about the external information, rather than confidence in the answer itself.
-
The uncertainty levels need stronger justification. In Fig. 2, lower stock does not necessarily mean greater uncertainty. For example, a 99% chance of stockout is more certain than a 50% chance, even though the risk is higher. The authors use expected consumption to explain why their high-uncertainty condition leaves both outcomes plausible, but this depends on assumptions about how much stock has been consumed since the last update. These assumptions need stronger validation. Without it, the experiments show how agents respond to changes in the evidence, but do not clearly isolate the effect of uncertainty itself.
-
Limited comparison with general information loss. RQ3 provides useful evidence that uncertainty-related qualifications disappear during communication. However, the paper does not compare their retention with that of other relevant information. Such a comparison would help determine whether these qualifications are especially vulnerable to loss or whether the observed behavior reflects a broader tendency to omit details during agent hand-offs.
Questions for authors
The problem is important, and the message-level findings are promising. However, the measurements and their interpretation do not clearly distinguish environmental uncertainty, confidence in an action, and information loss during communication. I would be inclined to raise my score if the authors adequately address or clarify these concerns.
4 comments
Dear Reviewer,
Thank you again for your great work on reviewing this paper submission to ICLR 2027.
The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.
Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.
Best regards, Your AC
Thank you to the authors for their response. For W3, I think it is better to compare the uncertainty with other secondary details unrelated to it (e.g., quantities, identifiers, or constraints), rather than with the fact it qualifies. The core fact is expected to survive a hand-off more often than any qualifier attached to it, so the gap between 85% and 2.5% does not show that uncertainty is more vulnerable than other details.
The response addresses my concerns on W1 and W2, so I have raised my score from 2 to 3.
We thank the reviewer for raising their score and for the suggested comparison, which we have now run.
W3: Agents keep every other detail of the case at least 5.7x as often as the uncertainty. We consider the same actions as before. On these actions, we check whether the agent restates details of the case that carry no uncertainty and that it has in context: four identifiers, a date, and three quantities.
| Kept after the first hand-off | Kind | Share |
|---|---|---|
| Serial number | Identifier | 62% |
| Purchase date | Date | 60% |
| Customer email | Identifier | 38% |
| Product price | Quantity | 27% |
| Package weight | Quantity | 25% |
| Registration ID | Identifier | 16% |
| Stock count | Quantity | 15% |
| Customer ID | Identifier | 14% |
| Uncertainty | 2.5% |
We added the table to App. F.3.4 (Table 17, L4050-L4068). While these facts are lost more frequently than the target concept with 85%, this evidence still suggests that secondary details unrelated to the uncertainty are lost more frequently.
Presentation: We have identified concrete issues and will fix them in the final version. Among others:
- Repeated and split explanations in Sections 2 and 3. What the levels order is explained twice, in S2 ("the levels order the risk by construction", L185) and in S3.1 ("how evenly the evidence splits [the branches]", L230-L231). The numerical and linguistic measurements are introduced in four places (L120-L122, L188-L196, L311-L312, L320-L323). We will explain each once.
- Terminology. Some terms are defined but not used later, such as the uncertainty path (L168-L169). We will remove such unused definitions and provide figures to visualize the remaining.
- Results. We will rewrite jargon-heavy sentences, such as "Slightly negative values are estimator bias rather than anti-persistence" (L407-L408), and define the judge measures of RQ3 before the example that uses them (L432-L437, defined only at L446-L459).
Rebuttal
We thank Reviewer UAmj for taking the time to read our manuscript and providing constructive feedback. We are glad they recognize that the problem is important, that our framework is useful and its message-based evaluation also applies to black-box systems, and that our analysis of the messages shows a practically meaningful failure.
We thank the reviewer for this important distinction. We now make it explicit in the main text, and a new ablation shows that it does not change our results on the uncertainty metrics. Section 2 (L188-L196) now defines the three constructs the reviewer distinguishes, uncertainty in the environment, confidence in an action, and loss in communication, together with their measures. The numeric metrics measure the confidence of an action, so a hedged action such as the reviewer's example can be stated confidently. For this reason, RQ3 reads the content of the actions: the Information Tracking Judge scores which outcomes an action still admits about the external fact (coverage) separately from how assertively it states them (concept assertiveness). The two diverge in our results. At medium and high, coverage falls to about 0.45 within five carrier actions, while concept assertiveness stays near a bare assertion at every level (S3.3, L445-L456). This matches prior work, in which models tend to not hedge even when uncertain (Yona et al., 2024; Zhou et al., 2024). To verify that the RQ1 result does not stem from confident hedged actions, we repeat it on the actions that commit to one outcome, for which the risk rises. The result holds. On these actions, the metrics respond to the level as weakly as on all actions of the origin agent, and every 95% interval of the difference includes zero (new App. F.1.4, L3607-L3617). The agents also recognize the uncertainty. No first action after the evidence omits it (S3.3, L472-L473), and when asked directly, each agent model rates the facts as more uncertain at high than at low in all nine families (App. D.9, L2453-L2457). The loss therefore lies mainly in carrying the uncertainty, not in recognizing it.
We now justify the levels in the main text and in a new per-family appendix, and an independent check with three LLMs supports their ordering. We thank the reviewer for the 99% example, which pinpoints what the levels order. They order how evenly the evidence splits the two possible outcomes (branches), not how far the evidence shifts from low. Low clearly favors one branch, and high leaves both branches plausible, so a 99% chance of stockout would count as less uncertain than our high level. We now state this in the main text with Figure 2 (S3.1, L230-L236), and the new appendix argues for each family why its levels order the uncertainty (App. C.2.3, L1793-L1910). In the reviewer's example, the reported stock leaves less room for consumption at each level, so stockout grows more likely under any consumption rate. The agents can estimate the consumption rate from the evidence itself, because the tool response that reports the stock also gives the count and time of the last refill (App. C.2.3). To validate these arguments, three frontier LLMs that do not serve as agents, blind to the level, estimated the probability of each branch for every task from the origin evidence and the records needed to read it (App. C.3, L1912-L1967). The judges form their own estimates, so the check does not rest on our consumption assumption. Each of them rates high as more uncertain than low in all nine families, and their mean uncertainty rises from low to medium in all nine.
We thank the reviewer for this suggestion and added the comparison, which shows that agents pass on a fact but drop its uncertainty. As a first comparison, we use information we already track, the fact that the uncertainty is about. This fact has the same relevance, the same path, and the same carriers as its uncertainty. After the first hand-off, at medium and high, 85% of the required carriers state the target concept, but only 2.5% keep both of its branches open (new App. F.3.4, L3953-L3971). This matches human communication, in which the claims survive but their hedges are lost (Fahnestock, 1986; Graeber et al., 2025). The uncertainty can only travel with its fact, so the fact caps how often the uncertainty is kept. At 85%, this cap lies far above the 2.5% we observe. The comparison therefore suggests that agents omit the uncertainty of a fact at higher rates than the fact itself.