acceptodds
Under review as a conference paper at ICLR 2027

Deliberation with Overlapping Evidence

Abstract

How should AI agents deliberate? Multi-agent AI systems increasingly improve their answers through deliberation, yet their agents often hold overlapping knowledge from searching the same corpora, calling the same tools, or sharing training data. This paper studies how AI agents should deliberate to reach the omniscient forecast: the Bayesian forecast given everyone's pooled private data. We first prove that exchanging posteriors (the belief protocol) cannot reach the omniscient forecast. We then propose the anecdote protocol, where agents share not only their beliefs but also the clues behind them, and prove that it reaches the omniscient forecast. Finally, we compare the two protocols with Bayesian and LLM agents and find that the anecdote protocol outperforms the belief protocol for both types of agents.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
multi agent debate, social learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1nApOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how two agents with overlapping evidence can combine their information through communication. The authors introduce a Bayesian model and show that exchanging beliefs alone may fail to reach the omniscient forecast, which uses all evidence held by both agents. They also show that randomly sharing clues and checking whether the receiver already knows them gives an error bound that decreases exponentially with the number of exchanges. The paper compares different clue-selection strategies through Bayesian simulations and controlled LLM experiments, using numerical clues and textual clues about a tennis match.

Critical strengths

The model is fairly clear and helps explain why overlapping evidence can limit information aggregation, and why agreement does not always mean that both agents’ information has been fully used. Under the stated assumptions, the authors provide an error lower bound for the belief protocol and a convergence bound for the anecdote protocol with random clue selection. These results give a quantitative comparison of the two protocols.

Critical weaknesses
  1. The writing and organization need more work, making the paper harder to read and check. The paper contains incorrect cross-references and several language errors, suggesting that it has not been carefully proofread. For example, Appendix A still contains an unresolved “??” on page 11, line 557. The main text refers to Appendix F in two places, but the relevant material is actually in Appendix E. The proofs in Appendix A also frequently refer back to earlier equations and lemmas, requiring readers to repeatedly look them up.

  2. The figures have clear problems with labeling and readability, making the experimental results harder to understand. For example, the caption of Figure 1 mentions “four experimental settings,” but the figure shows only three. Some panels use labels such as “Experiment E” and “Experiment F” without clearly explaining how they match the experiments in the main text. The text in the figures is also small, and many curves and shaded confidence intervals in Figure 1 overlap, making them hard to distinguish.

  3. The experimental evaluation is limited and simplified. The LLM mainly follows fixed instructions to select evidence and judge its effects, while a program computes the predictions. There is no comparison with an approach where LLMs directly exchange reasons. In addition, the text experiment uses only one tennis scenario with 12 facts, and repeated runs mainly change the presentation order. The range of tasks covered by the evaluation is therefore very limited.

Questions for authors
  1. Which strategies in Figure 1 perform better than others? Given the overlapping curves and confidence intervals, how can we tell whether these differences reflect a consistent advantage or random variation?

  2. If the theoretical error approaches zero, do the probability distance and disagreement in Figure 1 also approach zero? Please explain how these two measures relate to the squared log-odds error used in the theory.

  3. Why were these particular numerical values chosen for the seven evidence categories? Were they based on data or chosen by hand? What supports treating the resulting output as a probability and using its full-information version as the “omniscient forecast”?

4 comments
  1. Writing, organization, and proof correctness

Reviewer concern: Incorrect cross-references, typographical errors, and unresolved “??” in Appendix A.

We have revised the manuscript to correct grammatical and formatting issues, improve the organization of the experimental sections, and clarify the main results. Regarding Appendix A, the proof of Theorem 3.2 should explicitly identify the argument establishing equality when agents' clue-set sizes are common knowledge. This clarifies the existing proof without modifying the theorem or its assumptions. These corrections improve readability while preserving the theoretical contributions.

  1. Figure labeling and readability

Reviewer concern: Figure 1 mentions four settings although only three are shown; panels and confidence intervals are difficult to read.

We have revised Figure 1 to distinguish the three settings: algorithmic Gaussian, LLM Gaussian, and LLM realistic-match experiments. Each has two panels showing distance to the omniscient forecast and inter-agent disagreement. We also updated the figures using expanded simulations and 95% bootstrap confidence intervals, improved labels and captions, and corrected the number of settings. These changes clarify strategy comparisons, particularly during intermediate communication rounds.

  1. Experimental scope

Reviewer concern: The textual experiment uses one 12-fact match scenario, with only presentation order randomized.

We agree that the realistic-match experiment is limited in scope. It uses one tennis-match scenario containing 12 facts, with 100 repetitions varying presentation order. These repetitions improve our assessment of variability within this scenario but do not represent independent matches or forecasting tasks. We have clarified this limitation and do not claim generalization across different real-world forecasting scenarios.

  1. Relative performance of strategies

Reviewer concern: Which policies outperform others when confidence intervals overlap?

Updated Figure 1 provides clearer comparisons using expanded simulations and 95% bootstrap confidence intervals. In the algorithmic Gaussian experiment, unique, impactful, and strongest show rapid error reductions, although their rankings vary across rounds. Differences among strategies are smaller in the LLM Gaussian experiment, while the realistic-match experiment shows largely similar convergence trajectories.

Our main finding is not that one strategy consistently dominates, but that communicating underlying evidence substantially improves information aggregation compared with belief-only exchange. We distinguish this improvement from the smaller, setting-dependent differences among selection strategies and do not interpret overlapping confidence intervals as establishing statistical significance.

  1. Theoretical and empirical convergence metrics

Reviewer concern: Does vanishing theoretical error imply convergence of the metrics in Figure 1?

The theoretical and empirical measures are directly related. Our theory establishes convergence in squared log-odds error. Since the logistic function has derivative bounded by 1/4, absolute posterior-probability error is at most one quarter of absolute log-odds error. By the triangle inequality, inter-agent disagreement is bounded by the sum of both agents' distances from the omniscient forecast. Thus, vanishing expected squared log-odds error implies convergence in expected probability distance and disagreement.

The Gaussian and realistic-match experiments illustrate this relationship, with different communication strategies approaching the full-information forecast at different speeds. However, the theoretical guarantees assume binary symmetric clues, while the experiments use different evidence distributions and selection rules. We therefore do not claim that the theoretical bounds directly characterize the empirical convergence rates.

  1. Seven qualitative evidence weights

Reviewer concern: Why were the seven log-odds increments chosen, and what justifies the omniscient forecast?

The increments (-0.90, -0.50, -0.20, 0, 0.20, 0.50, 0.90) were manually specified, not calibrated from historical tennis data. Their symmetric mapping represents evidence direction and qualitative strength, with larger absolute values indicating stronger evidence.

The fixed mapping ensures consistent aggregation across strategies, isolating the effect of clue selection. These weights are heuristic rather than empirically calibrated likelihood ratios. Consequently, the resulting forecasts are outputs of the specified scoring model, not calibrated real-world win probabilities.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Thank you for the response. The explanations of the error metrics and the seven evidence weights clarified my questions. The reported changes to the writing and figures also address my comments. However, I remain concerned about the limited scope of the experiments. The current results do not provide enough support for broader tasks or more natural discussions between agents. Based on these clarifications and changes, I have slightly increased my rating, but I believe the paper still has room for improvement.

Thank you for the follow-up and for highlighting the limited range of textual tasks.

To address this concern, we added a second controlled textual experiment in a different domain: a hypothetical financial forecast of whether Company A will outperform Company B over three months.

We preserved the tennis experiment's structure (12 facts; 6/8 facts per agent with 2 shared; the same communication strategies, six rounds, seven-level LLM evidence classification, Python posterior aggregation, and 100 presentation-order repetitions).

In the new results, belief-only exchange remains flat, whereas clue-sharing reduces both distance to the scoring model's full-information forecast and inter-agent disagreement. The qualitative pattern is similar to the tennis experiment.

The revised paper includes the financial experiment in Section 4.3.2, with the full fact allocation and plots in Appendix F (Table 3 and Figure 3).

This expands the textual evaluation beyond a single sports scenario. We agree, however, that two controlled examples do not establish generalization across many real-world tasks or autonomous, free-form LLM debate. The finance facts are illustrative, and the internal benchmark is not a calibrated market forecast.

Reviewer_8PRYOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper studies information aggregation among AI agents with partially overlapping evidence. It proves that exchanging posterior beliefs alone may fail to recover the full-information Bayesian forecast, and proposes an anecdote protocol in which agents share individual clues and indicate whether the receiver already possesses them. Under the paper’s theoretical assumptions, this protocol converges to the full-information forecast with exponentially decreasing error.

The authors evaluate the approach using Bayesian agents with Gaussian signals, LLM agents with Gaussian signals, and LLM agents reasoning over textual evidence about a tennis match. Across these settings, clue-sharing strategies generally outperform belief-only communication in forecast accuracy and inter-agent agreement. However, the evaluation does not compare against one-shot or batch evidence sharing, nor does it quantify the communication cost and latency of iterative disclosure, leaving the practical efficiency and applicability of the proposed protocol insufficiently established.

Critical strengths
  1. Rigorous theoretical analysis. The paper formalizes deliberation with overlapping evidence and establishes a meaningful separation between belief-only communication and clue-sharing protocols: the former can have a nonzero error floor, while the latter achieves exponentially decreasing error under specified assumptions. This provides a principled explanation for why communicating evidence can be more effective than exchanging beliefs alone.

  2. Validation across multiple agent settings. The authors evaluate Bayesian agents and LLM agents using both Gaussian signals and textual evidence. Comparing multiple clue-selection strategies provides useful evidence that the benefits of evidence sharing extend beyond a single communication heuristic or synthetic setting.

Critical weaknesses
  1. Limited novelty and insufficiently justified problem formulation. The anecdote protocol—sharing individual clues and checking whether the receiver already possesses them—is intuitive and relatively straightforward. The paper proves its effectiveness under a specific overlapping-evidence model, but does not establish why sequential, one-clue-at-a-time communication is necessary or preferable to simpler alternatives such as batch sharing or exchanging clue identifiers. The theoretical guarantees are therefore tied to the chosen protocol and assumptions, while the motivation for these constraints remains insufficiently justified. It is unclear what fundamental new insight this work contributes beyond formalizing the intuition that sharing evidence helps resolve information overlap.

  2. Insufficient validation of practical benefits. The experiments primarily compare clue-sharing strategies against belief-only communication, without evaluating one-shot or batch evidence-sharing baselines. Moreover, the textual LLM experiment uses a synthetic scenario with manually defined mappings from qualitative evidence categories to log-odds increments, while posterior updates are computed externally. The results therefore do not establish that the proposed protocol improves end-to-end LLM reasoning or offers a favorable trade-off between accuracy, communication cost, and latency. This substantially limits the evidence for its practical value to the multi-agent research community.

Questions for authors
  1. What motivates the one-clue-at-a-time communication constraint? How does the proposed protocol compare with one-shot or batch evidence sharing, or exchanging clue identifiers to detect overlap, under comparable communication budgets? Such comparisons are important to establish whether the proposed protocol offers benefits beyond the intuitive idea of sharing evidence.

  2. In the textual LLM experiments, the LLM selects clues, while posterior updates are computed externally using a fixed mapping from qualitative evidence categories to log-odds increments. To what extent do the reported improvements reflect the communication protocol itself rather than this externally specified aggregation procedure?

5 comments
AuthorsOpenReview ↗

Response to reviewer 8PRY

  1. Motivation and novelty of one-at-a-time disclosure

Reviewer concern: Why sequentially transmit individual clues instead of batch sharing, exchanging clue identifiers or fully disclosing evidence?

Sharing all clues simultaneously would allow agents to recover the full information set immediately, assuming unrestricted communication and identifiable clues. However, this would not allow us to study how different clue-selection strategies affect the rate at which agents approach the omniscient forecast.

By restricting communication to one clue at a time, we can compare strategies such as strongest, random, and receiver-aware selection under a common sequential communication framework.

We do not claim that sequential sharing is more efficient than unrestricted batch disclosure. Rather, our objective is to understand which information agents should prioritize when communication is incremental or constrained.

  1. Practical efficiency: communication costs and latency

Reviewer concern: No one-shot/batch baseline or accounting of disclosure cost and end-to-end latency is reported.

In our experiments, belief-only communication transmits posterior beliefs, whereas anecdote communication additionally transmits a clue identifier, its content, and a membership response. Consequently, anecdote messages can require more communication per round.

Our experiments compare the effectiveness of different communication strategies over successive rounds, rather than their performance under identical bit or token budgets. The results therefore demonstrate the benefits of communicating underlying evidence under the specified protocol, but do not establish that anecdote sharing is more communication-efficient than belief exchange.

We have clarified this distinction and acknowledge matched-budget comparisons as an important direction for future evaluation.

  1. Source of improvements in LLM experiments

Reviewer concern: The LLM chooses facts while externally specified mappings and Python calculate posterior updates; how much improvement is due to the communication protocol?

We agree that posterior updates are computed externally, but would like to clarify the purpose of this design.

The Bayesian aggregation procedure is held fixed across communication strategies. Consequently, differences in their performance arise from which evidence is selected and communicated, rather than from differences in posterior-computation methods.

This separation is intentional: it allows us to evaluate evidence-selection policies without introducing additional variation from the LLM’s numerical reasoning capabilities.

In the realistic-match setting, the LLM also interprets the textual evidence before its contributions are aggregated. Therefore, the experiment evaluates LLM-guided evidence interpretation and communication under a controlled updating mechanism, rather than fully autonomous multi-agent reasoning.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

Thank you for the clarifications and revisions. I appreciate the authors' efforts to better position the contribution and extend the experiments.

However, my main concern remains unresolved. Given the assumption that clues have identifiable IDs, agents could simply exchange their clue ID sets to identify the overlap, without sequentially disclosing the actual clue contents. Together with the initial belief exchange, this seems sufficient to recover the omniscient forecast under the paper's setting. The authors do not explain why this simpler alternative is insufficient or compare against it.

The additional financial experiment is a useful extension, but it remains a single hypothetical scenario and does not substantially address the concerns about generalization or end-to-end LLM deliberation.

Overall, I appreciate the revisions, but my fundamental concerns about the necessity and practical contribution of the proposed protocol remain. I therefore maintain my original rating.

Reviewer_ipkrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Multi-agent systems increasingly have several models discuss a question before answering, and the designers of those systems assume discussion helps. Classical theory says something awkward: if rational agents just tell each other what they believe, they converge quickly, and when their evidence is independent they converge on the right answer. So why would explaining your reasons add anything? The authors propose that the answer lies in overlap. Two agents who read the same news report, retrieved the same document, or trained on the same text share evidence without knowing it, and beliefs alone cannot reveal which evidence is shared. The aspiration is a formal account of why “show your evidence” beats “state your belief” and of how fast each gets agents to the forecast they would reach by pooling everything. The author(s) defined a model in which agents hold random, partially overlapping subsets of clues, and proved two things. Exchanging beliefs, however many rounds, cannot get past a floor set by how much shared evidence remains ambiguous. Exchanging clues one at a time, with the receiver saying whether it already had that clue, removes the ambiguity and drives error down (geometrically). They then ran simulations with ideal Bayesian agents and with small language models choosing which clue to share, finding that belief-only exchange stalls while clue-sharing converges, with “share your strongest clue” among the fastest rules. The contribution is the clear statement of the problem and the impossibility result. The experiments largely confirm what the theorems already guarantee.

Critical strengths

The model feels/seems crisp, the example is memorable, and Remark 3.4 (that any index-free protocol inherits the floor) is the most useful statement in the paper. I liked (and largely agreed with) the link to agreement theorems and correlation neglect.

Critical weaknesses
  • The theory feels slight. Once Lemma 3.6 shows that belief messages reveal only counts, the floor is immediate, and the anecdotal result says that announcing clues reveals overlap. The novelty relative to Babichenko & Garber and Palley & Soll is the random allocation, which the paper should probably argue for rather than note.

  • The “LLM agents” currently do not deliberate. The LLM picks a lettered option under a stated strategy. Python does every belief update. The experiments test whether a 20B model can follow “send the strongest clue,” not whether LLM deliberation benefits from anecdotes.

  • Every anecdote variant converges by construction once all clues are announced, so Figure 1 largely shows that revealing everything reveals everything (duh). The interesting quantity is rate, and the paper currently never analyzes it. The (1−1/k)^t bound in Cor. 3.9 feels loose.

  • Communication cost is unmatched: anecdotes send more bits per round than beliefs.

  • The membership part assumes agents can recognize a shared document. In retrieval and training-data overlap, the motivating cases, that is precisely what is unknown. No?

  • Finish: a “??” in the proof of Theorem 3.2 (!) “how how” in 3.2, an unfinished sentence in 1.1, no results text for Section 4, and an empty page 8.

Rating. Weak reject. Presentation 2, informativeness 2, soundness 2 (the “??” and loose bound are fixable).

Questions for authors
  • What breaks if clues are conditionally dependent or if the existence of overlap is itself uncertain?

  • Can you match communication budgets (between anecdotes/tasks)?

  • Can you give a tight(er) rate for the strongest-clue rule?

  • Why does the sender ignore the signaling value of its own selection?

2 comments
AuthorsOpenReview ↗

Response to reviewer IPKR

  1. Novelty and relation to prior work: The closeness to classical models strengthens our impossibility result. Under fixed, commonly known allocations (Morris & Shin, 2002; Palley & Soll, 2019; Babichenko & Garber, 2021), belief exchange recovers the omniscient forecast. We formalize this contrast in the new Proposition 3.3. Introducing uncertainty about clue identities, without fundamentally changing the information structure, produces the persistent error floor in Theorem 3.2. Our contribution isolates this uncertainty as an obstacle to information aggregation.

  2. LLM deliberation: Our two LLM settings serve different purposes. In the Gaussian experiment, the LLM selects numerical clues while Python performs Bayesian updates. In the realistic-match experiment, the LLM additionally interprets natural-language evidence, classifying facts into seven qualitative strength categories before selecting which to communicate. Python then aggregates their numerical contributions consistently across strategies. Thus, the experiments evaluate LLM-guided evidence interpretation and communication, although not fully autonomous LLM deliberation. We have clarified this distinction in Section 4.3.

  3. Convergence rates and Figure 1: We agree that intermediate-round performance matters more than eventual convergence after complete disclosure. We updated Figure 1 using expanded simulations and 95% bootstrap confidence intervals. In the algorithmic Gaussian setting, unique, impactful, and strongest generally reduce error quickly, though rankings vary by round. Differences among strategies are smaller in the LLM Gaussian setting and largely overlap in the realistic-match setting. The contribution is therefore the comparison of information aggregation before full disclosure, not merely eventual convergence. Corollary 3.10 bounds random selection; we do not claim tighter theoretical rates for other strategies.

  4. Communication costs: We agree that equal rounds do not imply equal communication budgets. Belief-only communication transmits posterior beliefs, whereas anecdote communication additionally includes a clue identifier, content, and membership response. Our experiments compare performance across rounds rather than matched bit or token budgets. They demonstrate improved information aggregation under the specified protocol, not superior communication efficiency at equal cost. We explicitly acknowledge this limitation in the revision.

  5. Recognizing shared evidence: Our framework assumes identifiable clues, allowing receivers to recognize previously observed evidence. This isolates the effect of communication strategy from duplicate detection. We acknowledge that identifying equivalent information across retrieved documents or training data is substantially harder. Our experiments therefore concern identifiable evidence overlap, not general semantic overlap detection.

  6. Presentation and proof: We corrected typographical and grammatical errors, clarified the experimental methodology, improved Figure 1 and its labels, and expanded the interpretation of results. We also corrected the unresolved reference in Appendix A's proof of Theorem 3.2. These revisions improve clarity without changing the theorem's stated assumptions or conclusion.

  7. Dependent clues and uncertain overlap: Uncertain overlap is already covered: Definition 2.6 includes both fixed-overlap allocation D(kA,kB,m) and random-overlap allocation D(kA,kB), where overlap size M may be zero. Theorem 3.2 and Corollary 3.4 cover this random-allocation setting.

Conditional dependence is a separate issue. The relabeling argument underlying Lemma 3.7 relies on permutation invariance, suggesting that aspects may extend beyond independent clues. However, our stated bounds use the conditional-independence assumptions and additive log-likelihood structure. We do not claim that these guarantees extend unchanged to arbitrary correlated evidence.

  1. Strongest-clue convergence rate: Corollary 3.10 provides an exponential error bound for uniform random clue selection, not a tight bound for strongest-clue sharing. Strongest prioritizes clues with larger absolute log-likelihood ratios, potentially improving early-round accuracy, as illustrated empirically. Establishing a tighter strategy-specific bound would require analyzing clue informativeness jointly with overlap and selection order. We make no unsupported theoretical-rate claim.

  2. Signaling through clue selection: Clue selection can convey information beyond the announced value. Selecting the smallest available index reveals the absence of lower indices; selecting the strongest clue reveals that remaining clues are no stronger. Our experiments intentionally omit this signaling and update only on communicated evidence to isolate its direct contribution. A fully Bayesian receiver could exploit selection-induced information, potentially changing posterior beliefs and convergence.

Dear Reviewer,

Thank you again for your great work on reviewing this paper submission to ICLR 2027.

The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.

Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.

Best regards, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.