- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a framework for evaluating multi-agent systems on low-probability events. It uses synthetic environments with exactly computable conditional probabilities to separately evaluate evidence sharing, probability forecasts, and action selection. The authors compare independent ensembling, ordinary communication, and source verification. They also examine private records, source dependencies, and the disclosure of structural information. A separate decision experiment compares programmatic decisions with LLM decisions using either the original forecast probability or the correct probability. The results show that ordinary communication reduces average forecast error in the tested tasks, while the additional benefit of source verification remains unclear. Forecast accuracy and action optimality also need to be evaluated separately.
Critical strengths
-
The paper is reasonably well written.
-
The framework distinguishes between evidence not reaching an agent, errors in probability calculation after receiving evidence, and incorrect action selection given a probability. This provides more detailed diagnostic information than reporting only final forecast accuracy.
Critical weaknesses
-
The novelty mainly comes from combining existing computational tools and experimental controls, but the paper does not clearly explain what unique capabilities it adds compared with existing multi-agent evaluation frameworks.
-
Duplicate sources, failures in information sharing, group conformity, and action errors are not specific to low-probability tasks. The paper lacks sufficient comparisons with ordinary-probability tasks and does not clearly show how these problems become worse as probabilities decrease.
-
The source-verification prompt does not show a consistent additional benefit, and the effect of replacing forecasts with correct probabilities on action loss remains unclear. An evaluation paper does not need to propose a new algorithm, but it should better demonstrate how its diagnostics can guide system design and produce measurable practical improvements.
-
The main experiment includes only two worlds per stratum. Simulations in the appendix show that, under sparse-spike distributions, the nominal 97.5% confidence intervals achieve only about 70%–75% coverage. Further statistical sensitivity analyses are needed.
Questions for authors
See weeknesses