- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the problem of correlated evidence and misleading consensus in multi-agent reasoning. It proposes Counterfactual Witness Exchange (CWE), which evaluates candidate answers by replaying the reasoning process on different evidence subsets. The method measures answer support through the maximum number of disjoint supporting subsets and uses a one-step query strategy to reduce uncertainty under limited inference budgets. Experiments on HotpotQA and 2WikiMultiHopQA with three model backbones show that CWE achieves 0.53 EM and 0.65 F1, outperforming repeated full-context inference while using fewer tokens. Additional experiments examine the contributions of disjoint-capacity scoring and witness.
Critical strengths
1.The paper addresses a relevant issue in multi-agent reasoning, where agreement among agents may simply reflect repeated use of the same evidence. The disjoint-witness formulation is a reasonable way to account for evidence overlap while preserving the complementary information needed for multi-hop reasoning.
2.The capacity formulation is relatively clean, with explicit lower and upper bounds and a theoretically justified one-step query strategy. The witness exchange mechanism is also interesting, as it allows previously identified supporting sets to be reconsidered when better combinations are found.
3.The experiments provide some encouraging results. CWE improves EM from 0.45 to 0.53 over repeated full-context inference while reducing token usage from 24.5k to 12.7k.
Critical weaknesses
1.The evaluation is limited to five evidence units (32 subsets), but it is not entirely clear how these units are selected from the original QA examples.
2.The improvement over Greedy is relatively small (0.03 EM/F1), with 95% confidence intervals touching zero. Table 4 also shows that Greedy slightly outperforms CWE at query budgets of two and four. Stronger comparisons with existing evidence-selection methods or a single-agent subset-replay baseline would help clarify where the gains come from.
3.The capacity metric relies on whether a subset reproduces an answer, rather than whether the evidence actually supports it. Disjoint passages may still share the same source or lead to correlated errors. This concern is reflected in the 15% error rate among certified selections. Also, Equation (7) only certifies a winner among discovered candidates, which limits the interpretation of the guarantee.
Questions for authors
1.Please clarify the complete evidence preparation procedure. Are the units selected using a retriever, sampled from the original context, or derived from ground-truth supporting passages? Do the five units necessarily contain the evidence required to answer the question?
2.The improvements over Greedy are modest, and Greedy outperforms CWE at the smaller budgets reported in Table 4. Can the authors explain why the proposed acquisition policy is less effective in these settings?
To Reviewer Pr9i
We thank the reviewer for the careful examination of Table 2. The anomalies you identified are real, and they extend beyond Table 2: in nearly all reported tables, F1 exceeds EM by exactly 0.120, and the results across datasets and backbones differ by constant offsets.
We traced the problem to how the tables were compiled. The results were transferred into the tables manually in a spreadsheet, and formula drag-fill and copy errors during this step overwrote the actual values.
We are withdrawing the submission. In future work, all tables will be generated directly by script from preserved per-record outputs, which will be released with the paper. We thank you for identifying this issue.