- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper studies how two agents with overlapping evidence can combine their information through communication. The authors introduce a Bayesian model and show that exchanging beliefs alone may fail to reach the omniscient forecast, which uses all evidence held by both agents. They also show that randomly sharing clues and checking whether the receiver already knows them gives an error bound that decreases exponentially with the number of exchanges. The paper compares different clue-selection strategies through Bayesian simulations and controlled LLM experiments, using numerical clues and textual clues about a tennis match.
Critical strengths
The model is fairly clear and helps explain why overlapping evidence can limit information aggregation, and why agreement does not always mean that both agents’ information has been fully used. Under the stated assumptions, the authors provide an error lower bound for the belief protocol and a convergence bound for the anecdote protocol with random clue selection. These results give a quantitative comparison of the two protocols.
Critical weaknesses
-
The writing and organization need more work, making the paper harder to read and check. The paper contains incorrect cross-references and several language errors, suggesting that it has not been carefully proofread. For example, Appendix A still contains an unresolved “??” on page 11, line 557. The main text refers to Appendix F in two places, but the relevant material is actually in Appendix E. The proofs in Appendix A also frequently refer back to earlier equations and lemmas, requiring readers to repeatedly look them up.
-
The figures have clear problems with labeling and readability, making the experimental results harder to understand. For example, the caption of Figure 1 mentions “four experimental settings,” but the figure shows only three. Some panels use labels such as “Experiment E” and “Experiment F” without clearly explaining how they match the experiments in the main text. The text in the figures is also small, and many curves and shaded confidence intervals in Figure 1 overlap, making them hard to distinguish.
-
The experimental evaluation is limited and simplified. The LLM mainly follows fixed instructions to select evidence and judge its effects, while a program computes the predictions. There is no comparison with an approach where LLMs directly exchange reasons. In addition, the text experiment uses only one tennis scenario with 12 facts, and repeated runs mainly change the presentation order. The range of tasks covered by the evaluation is therefore very limited.
Questions for authors
-
Which strategies in Figure 1 perform better than others? Given the overlapping curves and confidence intervals, how can we tell whether these differences reflect a consistent advantage or random variation?
-
If the theoretical error approaches zero, do the probability distance and disagreement in Figure 1 also approach zero? Please explain how these two measures relate to the squared log-odds error used in the theory.
-
Why were these particular numerical values chosen for the seven evidence categories? Were they based on data or chosen by hand? What supports treating the resulting output as a probability and using its full-information version as the “omniscient forecast”?
4 comments
Dear Reviewer,
Thank you again for your great work on reviewing this paper submission to ICLR 2027.
The authors have now submitted their rebuttal. The open reviewer-author discussion phase will end soon, on November 18 at 11:59 PM AoE.
Please read the authors' responses, determine whether any further clarification is needed, and respond to the rebuttal at your earliest convenience. Please be sure to acknowledge the authors' responses and update your review accordingly.
Best regards, Your AC
Thank you for the response. The explanations of the error metrics and the seven evidence weights clarified my questions. The reported changes to the writing and figures also address my comments. However, I remain concerned about the limited scope of the experiments. The current results do not provide enough support for broader tasks or more natural discussions between agents. Based on these clarifications and changes, I have slightly increased my rating, but I believe the paper still has room for improvement.
Thank you for the follow-up and for highlighting the limited range of textual tasks.
To address this concern, we added a second controlled textual experiment in a different domain: a hypothetical financial forecast of whether Company A will outperform Company B over three months.
We preserved the tennis experiment's structure (12 facts; 6/8 facts per agent with 2 shared; the same communication strategies, six rounds, seven-level LLM evidence classification, Python posterior aggregation, and 100 presentation-order repetitions).
In the new results, belief-only exchange remains flat, whereas clue-sharing reduces both distance to the scoring model's full-information forecast and inter-agent disagreement. The qualitative pattern is similar to the tennis experiment.
The revised paper includes the financial experiment in Section 4.3.2, with the full fact allocation and plots in Appendix F (Table 3 and Figure 3).
This expands the textual evaluation beyond a single sports scenario. We agree, however, that two controlled examples do not establish generalization across many real-world tasks or autonomous, free-form LLM debate. The finance facts are illustrative, and the internal benchmark is not a calibrated market forecast.
Reviewer concern: Incorrect cross-references, typographical errors, and unresolved “??” in Appendix A.
We have revised the manuscript to correct grammatical and formatting issues, improve the organization of the experimental sections, and clarify the main results. Regarding Appendix A, the proof of Theorem 3.2 should explicitly identify the argument establishing equality when agents' clue-set sizes are common knowledge. This clarifies the existing proof without modifying the theorem or its assumptions. These corrections improve readability while preserving the theoretical contributions.
Reviewer concern: Figure 1 mentions four settings although only three are shown; panels and confidence intervals are difficult to read.
We have revised Figure 1 to distinguish the three settings: algorithmic Gaussian, LLM Gaussian, and LLM realistic-match experiments. Each has two panels showing distance to the omniscient forecast and inter-agent disagreement. We also updated the figures using expanded simulations and 95% bootstrap confidence intervals, improved labels and captions, and corrected the number of settings. These changes clarify strategy comparisons, particularly during intermediate communication rounds.
Reviewer concern: The textual experiment uses one 12-fact match scenario, with only presentation order randomized.
We agree that the realistic-match experiment is limited in scope. It uses one tennis-match scenario containing 12 facts, with 100 repetitions varying presentation order. These repetitions improve our assessment of variability within this scenario but do not represent independent matches or forecasting tasks. We have clarified this limitation and do not claim generalization across different real-world forecasting scenarios.
Reviewer concern: Which policies outperform others when confidence intervals overlap?
Updated Figure 1 provides clearer comparisons using expanded simulations and 95% bootstrap confidence intervals. In the algorithmic Gaussian experiment, unique, impactful, and strongest show rapid error reductions, although their rankings vary across rounds. Differences among strategies are smaller in the LLM Gaussian experiment, while the realistic-match experiment shows largely similar convergence trajectories.
Our main finding is not that one strategy consistently dominates, but that communicating underlying evidence substantially improves information aggregation compared with belief-only exchange. We distinguish this improvement from the smaller, setting-dependent differences among selection strategies and do not interpret overlapping confidence intervals as establishing statistical significance.
Reviewer concern: Does vanishing theoretical error imply convergence of the metrics in Figure 1?
The theoretical and empirical measures are directly related. Our theory establishes convergence in squared log-odds error. Since the logistic function has derivative bounded by 1/4, absolute posterior-probability error is at most one quarter of absolute log-odds error. By the triangle inequality, inter-agent disagreement is bounded by the sum of both agents' distances from the omniscient forecast. Thus, vanishing expected squared log-odds error implies convergence in expected probability distance and disagreement.
The Gaussian and realistic-match experiments illustrate this relationship, with different communication strategies approaching the full-information forecast at different speeds. However, the theoretical guarantees assume binary symmetric clues, while the experiments use different evidence distributions and selection rules. We therefore do not claim that the theoretical bounds directly characterize the empirical convergence rates.
Reviewer concern: Why were the seven log-odds increments chosen, and what justifies the omniscient forecast?
The increments (-0.90, -0.50, -0.20, 0, 0.20, 0.50, 0.90) were manually specified, not calibrated from historical tennis data. Their symmetric mapping represents evidence direction and qualitative strength, with larger absolute values indicating stronger evidence.
The fixed mapping ensures consistent aggregation across strategies, isolating the effect of clue selection. These weights are heuristic rather than empirically calibrated likelihood ratios. Consequently, the resulting forecasts are outputs of the specified scoring model, not calibrated real-world win probabilities.