acceptodds
Withdrawn from ICLR 2027

Counterfactual Witness Exchange for Evidence-Grounded Multi-Agent Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Repeated model calls can produce agreement without adding evidence for an answer. In evidence-grounded multi-hop reasoning, several passages may be useful only in combination, while different supporting sets can overlap. We hypothesize that accounting for this structure can improve answer selection and inference allocation. We propose Counterfactual Witness Exchange (CWE), which queries evidence subsets by replaying the complete answering process, including subsequent peer exchanges. CWE measures support capacity as the maximum number of pairwise disjoint nonempty subsets that reproduce an answer under a fixed response rule, assigning zero capacity to answers supported without evidence. Lower and upper capacity bounds guide queries toward unresolved comparisons, while newly observed answers enter the candidate set. A separate bound for undiscovered answers prevents certification based solely on an incomplete candidate menu. The resulting certificate concerns capacity under a fixed replay function, not factual correctness. Existing communication and matched-text diagnostics motivate this distinction; the nominal-label effect is concentrated in one tested backbone. Our evaluation framework separates answer quality, support structure, and query efficiency through matched-budget comparisons, shared-observation aggregation controls, and structural interventions. It is designed to test when alternative support discovery makes additional reasoning useful.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, multi-agent reasoning, evidence-grounded reasoning, counterfactual inference, test-time computation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_o4z4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the problem of correlated evidence and misleading consensus in multi-agent reasoning. It proposes Counterfactual Witness Exchange (CWE), which evaluates candidate answers by replaying the reasoning process on different evidence subsets. The method measures answer support through the maximum number of disjoint supporting subsets and uses a one-step query strategy to reduce uncertainty under limited inference budgets. Experiments on HotpotQA and 2WikiMultiHopQA with three model backbones show that CWE achieves 0.53 EM and 0.65 F1, outperforming repeated full-context inference while using fewer tokens. Additional experiments examine the contributions of disjoint-capacity scoring and witness.

Critical strengths

1.The paper addresses a relevant issue in multi-agent reasoning, where agreement among agents may simply reflect repeated use of the same evidence. The disjoint-witness formulation is a reasonable way to account for evidence overlap while preserving the complementary information needed for multi-hop reasoning.

2.The capacity formulation is relatively clean, with explicit lower and upper bounds and a theoretically justified one-step query strategy. The witness exchange mechanism is also interesting, as it allows previously identified supporting sets to be reconsidered when better combinations are found.

3.The experiments provide some encouraging results. CWE improves EM from 0.45 to 0.53 over repeated full-context inference while reducing token usage from 24.5k to 12.7k.

Critical weaknesses

1.The evaluation is limited to five evidence units (32 subsets), but it is not entirely clear how these units are selected from the original QA examples.

2.The improvement over Greedy is relatively small (0.03 EM/F1), with 95% confidence intervals touching zero. Table 4 also shows that Greedy slightly outperforms CWE at query budgets of two and four. Stronger comparisons with existing evidence-selection methods or a single-agent subset-replay baseline would help clarify where the gains come from.

3.The capacity metric relies on whether a subset reproduces an answer, rather than whether the evidence actually supports it. Disjoint passages may still share the same source or lead to correlated errors. This concern is reflected in the 15% error rate among certified selections. Also, Equation (7) only certifies a winner among discovered candidates, which limits the interpretation of the guarantee.

Questions for authors

1.Please clarify the complete evidence preparation procedure. Are the units selected using a retriever, sampled from the original context, or derived from ground-truth supporting passages? Do the five units necessarily contain the evidence required to answer the question?

2.The improvements over Greedy are modest, and Greedy outperforms CWE at the smaller budgets reported in Table 4. Can the authors explain why the proposed acquisition policy is less effective in these settings?

Reviewer_FAh4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that agreement among agents in multi-agent QA can repeat the same evidence, and proposes Counterfactual Witness Exchange (CWE) for answer selection. A witness is a nonempty evidence subset that reproduces an answer when five agents restart from only that subset; an answer's support capacity is the maximum number of witnesses that share no passage. CWE keeps lower and upper bounds on each candidate's capacity and chooses the next subset to replay with a query rule that is optimal for the worst-case one-step reduction in total interval width (Theorem 1). Packings are recomputed after each replay, so an earlier witness can be exchanged for several compatible ones. Experiments on 500 HotpotQA and 500 2WikiMultiHopQA questions with Qwen2.5-7B, Llama-3.1-8B and Mistral-7B (9,000 records per method) compare CWE with repeated full-context inference and several subset-query policies, and add fixed-observation, query-restriction and structural experiments to separate scoring from search.

Critical strengths
  1. The mechanism decomposition is solid. With the replay observations fixed and only the scoring rule changed, capacity scoring gives higher EM than counting supporting subsets (.53 vs .51 on CWE prefixes, .45 vs .44 on random prefixes). The matched query-restriction experiment tests witness exchange directly: CWE gains .09 EM over the restricted policy when the initial witness blocks alternatives and .02 when it does not, an interaction of .07 [.02, .12]. These experiments connect the gain to the two parts of the method rather than to extra inference alone.

  2. The paper reports its failure modes honestly. Certified records still have a conditional error rate of .15 (675 of 4,500), and the empty-control rule itself costs accuracy: rescoring without it gives a net EM gain of .018 [.008, .028]. This makes the limits of the support score visible.

Critical weaknesses
  1. The gain over the strongest baseline is not significant. Against greedy, the EM gain is .03 [.00, .06], and the paper states that the interval includes zero; significance holds only against the weaker repeated full-context baseline (.08 [.05, .11]). CWE has the higher point estimate in all six dataset–backbone cells, but at allowances two and four it is below greedy (.45 vs .46 and .47 vs .48), and allowance four is the comparison the paper describes as isolating query choice at a similar realized cost.

  2. The scale is limited. Packing and sensitivity evaluation use exact subset enumeration, which is exponential in the number of evidence units, and all experiments use five-unit contexts (32 subsets) with 7–8B models. Larger contexts and open retrieval settings are not tested, so the evidence does not support the generality of the claim.

Questions for authors
  1. (Clarification) How are the five evidence units chosen from the ten paragraphs of each HotpotQA and 2WikiMultiHopQA item, and do they always include the gold paragraphs? I could not find this in the paper, and it affects how hard and how general the setting is.

  2. (Clarification) Can you report paired intervals for CWE against greedy at allowance four, where realized costs match, and for the scoring comparison in Table 5 (.53 vs .51)?

  3. (Suggestion, future work) Have you tried an approximate packing or sampled subsets for larger contexts, for example all ten paragraphs (1,024 subsets)? Even a small result would show whether the method scales beyond five units.

Reviewer_Pr9iOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper addresses the issues of evidence reuse and false consensus in multi-agent reasoning by proposing the CWE method, which evaluates the support for an answer through counterfactual replay and disjoint evidence sets. While the research topic is significant, the proposed method lacks sufficient innovation, and the experimental validation is notably inadequate; the current experiments are insufficient to substantiate the paper's core conclusions.

Critical strengths

无

Critical weaknesses

In Table 2, the values ​​in the upper section all end in 0, while those in the lower section all end in 5. Furthermore, the difference between the F1 and EM scores is consistently 0.120 across all models and datasets; I find this data to be implausible.

Questions for authors

In the upper section of Table 2, the final digits of all data entries are 0, while in the lower section, the final digits are all 5.

In Appendix Table 2, the difference between the F1 score and the EM (Exact Match) score is 0.120 across all models and datasets. Please provide the raw prediction results and the complete calculation process to explain this phenomenon.

1 comment
AuthorsOpenReview ↗

To Reviewer Pr9i

We thank the reviewer for the careful examination of Table 2. The anomalies you identified are real, and they extend beyond Table 2: in nearly all reported tables, F1 exceeds EM by exactly 0.120, and the results across datasets and backbones differ by constant offsets.

We traced the problem to how the tables were compiled. The results were transferred into the tables manually in a spreadsheet, and formula drag-fill and copy errors during this step overwrote the actual values.

We are withdrawing the submission. In future work, all tables will be generated directly by script from preserved per-record outputs, which will be released with the paper. We thank you for identifying this issue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.