acceptodds
Withdrawn from ICLR 2027

From Evidence to Action: MAS Prediction and Decision for Low-Probability Events

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Predicting low-probability events requires both preserving the order of magnitude of probabilities and identifying evidence dependencies and executing actions aligned with the specified loss objective. Existing evaluations of Multi-Agent System (MAS) prediction and collaboration rarely distinguish communication gains, information propagation, and action execution within a single task, and building agents from the same foundation model further limits what can be inferred about cross-model collaboration. We propose a paired framework benchmarked against computable conditional probabilities. In a virtual public-service scenario, we manipulate source relationships and the disclosure of private records and structural information, and use a new decision branch to separate probability substitution from action execution. We use four models to form Homogeneous MAS, which jointly constitute a Hybrid MAS where each virtual world uses four balanced compositions and two decoding runs, and all experiments are repeated on the same worlds. In the main experiment over 128 worlds, ordinary communication relative to independent ensembling yields a mean absolute error difference of in the Homogeneous MAS and in the Hybrid MAS. After two rounds of communication, the availability rate of valid private raw records for ordinary communication and source verification in the Hybrid MAS approaches 84%. Through paired replication across compositions, verifiable information channels, and public loss objectives, this study shows that group consensus, probability accuracy, and action optimality require separate evaluation benchmarks.

Primary area:
interpretability and explainable AI
Keywords:
Multi-Agent Systems, Heterogeneous Models, Low-Probability Prediction, Evidence Dependency, Probabilistic Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_JnoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework for evaluating multi-agent systems on low-probability events. It uses synthetic environments with exactly computable conditional probabilities to separately evaluate evidence sharing, probability forecasts, and action selection. The authors compare independent ensembling, ordinary communication, and source verification. They also examine private records, source dependencies, and the disclosure of structural information. A separate decision experiment compares programmatic decisions with LLM decisions using either the original forecast probability or the correct probability. The results show that ordinary communication reduces average forecast error in the tested tasks, while the additional benefit of source verification remains unclear. Forecast accuracy and action optimality also need to be evaluated separately.

Critical strengths
  1. The paper is reasonably well written.

  2. The framework distinguishes between evidence not reaching an agent, errors in probability calculation after receiving evidence, and incorrect action selection given a probability. This provides more detailed diagnostic information than reporting only final forecast accuracy.

Critical weaknesses
  1. The novelty mainly comes from combining existing computational tools and experimental controls, but the paper does not clearly explain what unique capabilities it adds compared with existing multi-agent evaluation frameworks.

  2. Duplicate sources, failures in information sharing, group conformity, and action errors are not specific to low-probability tasks. The paper lacks sufficient comparisons with ordinary-probability tasks and does not clearly show how these problems become worse as probabilities decrease.

  3. The source-verification prompt does not show a consistent additional benefit, and the effect of replacing forecasts with correct probabilities on action loss remains unclear. An evaluation paper does not need to propose a new algorithm, but it should better demonstrate how its diagnostics can guide system design and produce measurable practical improvements.

  4. The main experiment includes only two worlds per stratum. Simulations in the appendix show that, under sparse-spike distributions, the nominal 97.5% confidence intervals achieve only about 70%–75% coverage. Further statistical sensitivity analyses are needed.

Questions for authors

See weeknesses

Reviewer_aLTMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a controlled evaluation framework for assessing the reliability of LLM-based multi-agent systems (MAS) in low-probability event forecasting and decision-making. The framework uses synthetic scenarios based on Bayesian networks, where ground-truth event probabilities and action losses can be computed, to separately evaluate three aspects: evidence propagation, probability prediction, and downstream action selection.

The authors compare independent ensembling, multi-agent communication, and source-verification protocols across homogeneous and heterogeneous MAS configurations. They further investigate the effects of evidence-source dependencies, private information sharing, and structural information disclosure.

Experimental results suggest that multi-agent communication generally improves average forecasting accuracy compared with independent ensembling, although greater agreement among agents does not necessarily lead to more accurate predictions. The authors also find that private evidence is not always successfully propagated among agents and that LLM-based action selection can incur additional expected loss compared with deterministic optimization, even when given the same probability. Overall, the paper argues that information propagation, forecasting accuracy, and decision optimality should be evaluated separately when assessing MAS reliability.

Critical strengths
  1. The paper introduces a controlled evaluation framework that separately examines evidence propagation, probability forecasting, and action selection in multi-agent systems. By using synthetic scenarios with computable ground-truth probabilities and explicit loss functions, the framework provides a systematic way to evaluate different aspects of MAS performance rather than relying solely on final prediction accuracy.

  2. The paper provides several interesting observations about multi-agent collaboration and decision-making. In particular, the experiments show that communication can improve average forecasting accuracy compared with independent ensembling, while greater agreement among agents does not necessarily imply more accurate predictions. The results also highlight the distinction between probability accuracy and decision optimality, as LLM-based action selection can incur additional expected loss even when supplied with the same probability.

Critical weaknesses
  1. As I understand it, the main contribution of this paper lies in separating evidence propagation, probability forecasting, and action selection within a unified evaluation framework. While I think this is a reasonable and potentially useful direction, I am not entirely convinced that the paper sufficiently demonstrates the additional value of this decomposition. Several findings, such as greater consensus not necessarily implying better accuracy and accurate probabilities not guaranteeing optimal decisions, are informative but not entirely new. I would have liked to see a deeper analysis of how this framework helps identify specific failure mechanisms or guide improvements to MAS. In particular, since the optimal action in the current setting can be directly determined from the probability and an explicit loss function, I am unsure what additional insights are gained by introducing another LLM for action selection, or how representative this setting is of more complex decision-making tasks.

  2. I think some of the main empirical findings would benefit from more detailed analysis and robustness checks. For example, the paper shows that average disagreement decreases while average prediction error increases, but does not clearly report how frequently both occur within individual worlds. Similarly, the action experiments report average expected regret without showing how often agents select suboptimal actions even when correctly using the ground-truth probability. These statistics would help clarify how representative and practically significant the reported phenomena are. I also have some concerns about the reliability of the results: Table 8 shows substantial undercoverage of the confidence intervals under certain distributions, while the validation and fallback mechanism may introduce additional errors unrelated to probability reasoning. Although the authors acknowledge these limitations, I think further analysis is needed to establish the robustness and interpretation of the main conclusions.

Questions for authors
  1. Could the authors further clarify the motivation for separating the evaluation into evidence propagation, probability forecasting, and action selection? I understand that this allows different aspects of MAS performance to be evaluated separately, but I wonder whether the framework could also demonstrate how such a decomposition helps improve MAS. For example, could the authors provide a case where identifying a failure at a particular stage leads to a targeted improvement that would be difficult to obtain from evaluating only the final outcome?

  2. As I understand it, separating probability estimation from action selection is useful for evaluating different stages of MAS performance. However, I wonder what additional insights this design provides compared with allowing agents to make decisions directly from the available evidence. Would it be possible to include a direct evidence-to-action baseline? Relatedly, given that the optimal action in the current setting can be calculated directly from the probability and loss function, could the authors further clarify the motivation for introducing an additional LLM decision step?

  3. The existing metrics provide useful information about the relationship between consensus and forecasting accuracy. However, I am more curious about how frequently agent disagreement decreases while prediction error actually increases. Could the authors report the proportion of individual worlds where this occurs? I think this would help clarify whether the phenomenon is relatively common or mainly driven by a small number of cases, as well as how substantial the accuracy degradation is.

  4. The expected regret metrics provide useful insights into the quality of action selection. However, I am more interested in how often LLMs actually make suboptimal decisions, particularly when the correct probability is provided. Could the authors report the proportion of such cases and further distinguish between failures to correctly use the supplied probability and failures to select the optimal action despite using it correctly? It would also be interesting to examine cases where correcting the probability changes the optimal action, since improvements in probability accuracy do not necessarily translate into different decisions.

  5. Could the authors provide additional analyses to clarify the robustness of the main findings? In particular, it would be useful to report the communication gains separately for the direct-probability and evidence-integration modes, and to examine whether the conclusions remain similar when probability forecasts are not replaced due to unrelated action-field validation failures, particularly in the structural-disclosure experiment. Given the coverage results in Table 8, I also wonder whether the actual world-level paired differences exhibit similar characteristics to the distributions under which the confidence intervals perform poorly, and whether alternative inference methods would lead to similar conclusions.

  6. As I understand it, the benchmark relies on synthetic scenarios where the ground-truth event probabilities can be computed exactly. I think this is a useful design for controlled evaluation. However, in real-world low-probability forecasting, the true probabilities are generally unknown, and forecasting performance is much harder to evaluate. I wonder to what extent the findings from this benchmark can generalize to realistic forecasting tasks. In particular, does the framework primarily evaluate the ability of LLMs to reason about probabilities under a known probabilistic model, rather than their ability to forecast rare events in more realistic settings? It would be helpful if the authors could further clarify the intended scope and limitations of the benchmark.

Reviewer_nDGMOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies an important problem of estimating the probability of tail events (low probability) in a particular multi-agent system (MAS) scenario.

Critical strengths

None.

Critical weaknesses

Unfortunately, I cannot understand this paper.

Without prejudice to the authors, from my experience and opinion, this paper has the hallmarks of AI slop.
The work jumps directly into the results without a strong understanding or introduction to the worlds and the definitions of the protocols and does not use significant precision appropriately.

As the start of a genuine review, before giving up:

  • You should separate a word from a leading parentheses for better readability. This seems to have been a global decision on the authors’ part.
Questions for authors

If this is genuine work, please consider reading through your work to help others understand what you are trying to communicate. The core idea, even in the abstract, is unclear.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.