acceptodds
Withdrawn from ICLR 2027

How Report Formation Shapes Evidence Use in Multi-Agent Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

13 views

Abstract

Giving a final language model every source document may seem to make evidence sharing among upstream agents redundant. We test this assumption by varying source access during report formation while fixing the final sources and the agents' initial answers and confidence values. Across 249 task variants from 128 synthetic decision scenarios, sharing during a report update improves final accuracy by 18.36 percentage points. We exchange naturally generated explanation records, statements relating cited passages, independently of their automatically copied source text. Explanations formed after sharing improve accuracy by 13.48 points under fixed copies. Thus sharing changes useful report content beyond the source passages explicitly repeated for the final model. We next retain the earlier report and supply a correct, reference-derived judgment of whether a rule applies to the current case. This diagnostic revision improves decisions, although the predicted reverse effect of leaving applicability unresolved is not established. Finally, both a correct and an unresolved interpretation remain present at matched positions. Designating the correct interpretation as the report's current judgment rather than an unincorporated external annotation improves accuracy by 9.18 points. Benchmark studies examine relevance under separately specified information controls. Together, these interventions show that complete final access leaves report formation consequential: generated explanations and their assigned report roles affect how available evidence is used.

Primary area:
causal reasoning
Keywords:
Multi-agent reasoning, large language models, evidence sharing, report formation, causal interventions, evidence use

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HHNDOpenReview ↗
Rating
3
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Does sharing between agents matter if the final model can read every source?

The controlled design: fixed initial answers, swapped components, pre-registered tests for the later experiments.

Critical strengths

controlled exp. sharing what to share adding correct note labels

over 2 datasets HiddenBench (65 questions) and M3GQA (800 questions

Critical weaknesses
  • temp is 0, thinking disabled yet real finding it gets is "pre-written reasoning helps a model that isn't allowed to reason", which is same as chain-of-thought
  • while sharing explanation it may contain yhe answer already
  • On HiddenBench when final model has all resources, report hurts with comparison, but on synthetic tasks it help (appendixD5) so synthetic setup favours while real tasks dont
  • team setup is fixed, which doesnt challenge how things would be like if extrapolated
  • wrong reports aren't tested: What happens when agents connect the dots incorrectly? Sharing could spread errors, which matters for safety
Questions for authors
Reviewer_JfsEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies whether information sharing between upstream agents is still useful when the final LLM already has access to all original sources. The authors show that source sharing improves final acc by 18.36% points even with fixed initial recommendations and complete source access. They also explore the roles of generated explanations, report revisions and interpretation. They have an interesting finding that upstream communication matters even with complete final source access. However, their experiments show that report content affects decisions but they provide very less evidence about why report content improves decisions. Even after few times reading it is still unclear to me.

Critical strengths
  • The paper addresses an interesting question in multi-agent LLM systems. Just because a model has access to the information doesn't mean it will use it correctly. I also liked their controlled experimental setup.

  • The authors conduct several intervention experiments rather than relying only on overall accuracy. The explanation-exchange and report-role experiments are interesting. Reporting confidence intervals and including HiddenBench and M3GQA evaluations also strengthens the work.

Critical weaknesses
  1. For the explanation mechanism(183-193), the authors exchange complete explanation collections. So there can be changes in wording, length, source links and restated facts together, right? The 13.48 point improvement is nice to see but I am not sure which of these factors is actually responsible for the gain. Also it is very difficult to connect this result directly to the original 18.36 point improvement from source sharing since the two experiments use different interventions.

  2. I don't think the preservation note used as the main baseline may be neutral. In Table 3, acc decreases from 74.61% to 70.12% without sharing. So the reported 27.34 point improvement over preservation may partly reflect a harmful baseline rather than only the benefit of the resolved revision. IMO it would be useful to test alternative neutral or length-matched notes to separate the effect of adding a revision from the effect of providing the correct judgment.

  3. Here the main motivation is that upstream information sharing helps the final model use evidence more effectively through better report formation. But if I look at the source only comparisons, I see a very different picture. For example, in the historical HiddenBench regenerated-report setting(Appendix D.5), source-only accuracy reaches 87.69% but the full-report condition achieves only 45.38% even after supporting-material exchange. Well this large performance gap actually raises a fundamental question: if reports help the model use evidence more effectively, why does adding reports lead to such a substantial drop in acc? I understand these are different experimental settings and the authors also acknowledge this limitation. But the paper does not sufficiently explain the difference. This matters a lot because your main experiments show that sharing improves performance within a report-based workflow but do not establish that using reports is better than directly providing the original sources. I think this issue deserves more analysis rather than being left mainly as a limitation.

  4. The main experiments rely on synthetic tasks and the benchmark evaluations use different controls so generalizability is still a concern. Also the revision experiment(Sec. 5.1) relies on ground-truth corrections rather than agent-generated ones.

  5. The introduction cites only three works. This makes me feel that the authors have not sufficiently reviewed the existing literature which also makes it harder to understand the novelty and positioning of the paper. For instance, [1,2,3] are very relevant to the questions studied here. Authors should include more relevant references throughout the paper to better support their claims and connect their findings to existing literature.

I am leaning toward weak rejection since the main mechanism is still unclear to me and the practical benefits need stronger evidence. Hopefully, the authors can clear up these concerns during the rebuttal.

[1] https://arxiv.org/pdf/2508.17536

[2] https://arxiv.org/pdf/2512.08296

[3] Liu et al. "Lost in the middle: How language models use long contexts." TACL, 2024.

Questions for authors
  1. Can the authors separate the effects of source links, explanation wording and repeated facts instead of exchanging complete explanation collections? Which of these actually contributes most to the improvement?
  2. Why does the source only condition substantially outperform full reports in some HiddenBench settings? What determines whether reports help or hurt the final model?
Reviewer_tyAbOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies whether upstream information sharing remains useful when the final model already receives all original sources. Controlled experiments show that sharing improves report quality and final accuracy, particularly by producing explanations that connect relevant evidence. The paper further finds that correct report revisions and the assigned role of an interpretation affect how the final model uses available evidence.

Critical strengths
  1. The paper explores an intriguing issue: even when the final model has access to all the original source material, the process by which upstream agents generate reports can still influence the ultimate decision.

  2. The experimental design is quite sound; by employing various controls, the authors effectively isolate the effects of different factors, thereby identifying the true underlying cause.

  3. In addition to the synthesis tasks, the paper presents supplementary experiments using HiddenBench, M3GQA, and several other models; the results generally support its core conclusions.

Critical weaknesses
  1. In fact, the abstract of this paper is quite difficult to understand. It is recommended that the authors revise the research questions, the motivation, and the way the methodology is presented. Clearly, the phrasing in the abstract sounds too much like it was generated by AI.

  2. Even if the final model has access to all sources, upstream sharing is still useful. However, sharing does more than just change the visibility of information; it also allows multiple upstream models to perform additional relationship extraction, rule matching, and case evaluation in advance. Therefore, the root cause lies not in the sharing itself, but in the additional computation.

  3. While this paper explores the necessity of sharing in the context of closed-source models, it lacks a deeper investigation—for instance, by utilizing open-source models and employing interpretability techniques to demonstrate that such sharing is both necessary and advantageous.

  4. The core controlled experiments primarily utilize a single DeepSeek model; there is insufficient evidence for cross-model generalization.

Questions for authors
  1. If the final model were given more inference time to establish observation-rule links on its own, would the benefits of upstream sharing disappear? This determines whether communication is irreplaceable or merely performs a portion of the computation on behalf of the final model.

  2. The explanations generated after ``Sharing'' may simply be expressions of the correct answers that the agents had already arrived at. How can we prove that the explanation itself drove the correct decision, rather than being a post-hoc trace left behind by correct internal reasoning?

  3. Does this phenomenon accumulate or diminish in longer agent chains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.