- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper tested a multi-agent system where several AI agents discuss a question, compare their reasons and reach a decision using a graph. The paper explores the problem that if using multiple reasoning by different AIs, with structured argumentation graphs, will this be helpful to identify valid claims and settle disagreements. Its main idea is to record which arguments support or challenge other arguments, then test how removing a connection changes the decision.
Critical strengths
-
Because the graph participates in decision computation, interventions can test their dependencies within the actual aggregation procedure, which naturally embeds the system with some explanations.
-
The interaction between different AIs itself is an interesting topic to explore.
Critical weaknesses
-
Why should the additional AI judge be trusted over the argument graph? When the two judgments disagree, what evidence shows that the additional judge is more likely to be correct? This matters because agreement with another AI is not necessarily agreement with reality.
-
Faithfulness to the graph’s computation does not establish faithfulness to the underlying LLM’s internal reasoning. A valuable follow-up would compare graph interventions with interventions on agent inputs and subsequent generations. A better-designed qualification for causal claims is in need.
-
Tree shaped argumentation and single edge interventions simplify analysis, interacting or redundant reasons may require joint interventions. A reason can matter collectively while showing little effect when removed individually.
Questions for authors
Are the two confidence scores meaningfully comparable? The correction rule compares confidence from graph calculations with confidence assigned by the AI judge. Both are normalised, but they come from different processes. arxiv.org For example, a graph score of 0.7 and a judge score of 0.8 may not represent the same kind of certainty. What evidence shows that a higher judge score means its answer is more reliable? Normalising numbers so they sum to one does not, by itself, make them equally trustworthy.
We thank the reviewer for their thorough review. We address each of the weaknesses and the questions below.
W1: Why should the additional judge be trusted over the argument graph?
It should not be trusted unconditionally, and ARGORA does not do so. Instead, we treat the observational judge as a fallback assessment, in the case where we believe that the argument graph’s decision is too fragile (or more precisely, override when a single edge perturbation already changes the graph’s own winner). This override additionally requires satisfaction of the winner-confidence gate, and improvement in the alignment–perturbation objective relative to leaving the argument graph unchanged (Definition 4.6). As for the performance of the judge itself, since it is also reliant on the ARGORA’s generated transcripts (without the argumentation graph), our evaluations have found it to have comparable performance to ARGORA as shown in Appendix G.9, which makes it a useful corrective signal.
W2: graph faithfulness vs the LLMs' internal reasoning
We actually considered this proposal while designing ARGORA. The main obstacle is computation / token cost, where a direct input intervention requires regenerating every downstream review, rebuttal and base score for each perturbed argument. This cost multiplies over every argument in every argumentation graph, so this work is restricted to fixed-graph interventions.
Our causal claims are scoped accordingly in the paper. The causal object is the aggregation procedure (Prop. 4.1): edge-local counterfactuals state exactly how the decision depends on each argumentative channel, not why an LLM wrote an argument or how it reasoned internally. The design is motivated by the unfaithfulness of free-form reasoning traces, where instead of trusting a trace to reflect latent reasoning, we make the decision depend on an explicit, recomputable object. We will clarify the scope based on our design further in our paper, and we do believe that building a causal system on the underlying LLM’s reasoning is out of scope of our work and leave it as a promising future research direction.
W3: Tree topology and single-edge interventions may miss joint reasoning.
We do agree that single-edge deletions do not capture every way reasons combine. Effects in a QBAF are not additive (App. E.2), so a reason with a small individual effect may matter jointly with others. That said, joint interventions are well-defined in our framework (Lemma C.12, Def. 4.6), where a joint intervention is a set of single-edge deletions applied together. Our reported diagnostics apply one deletion at a time.
Other than that, the tree topology is a deliberate consequence of ARGORA’s discussion protocol, detailed in Appendix D. Each main argument anchors a separate discussion, and each supplementary argument therefore has an explicitly designated parent. We’d like to also clarify that redundancy is handled at construction, using contextual orthogonality pruning (Sec. 4.2, App. D.4): near-duplicate siblings are removed before evaluation so that single deletions stay informative. We went ahead with this design choice as we found that unstructured and unguided discussion processes can lead to endless spirals of circular reasoning or experts going back and forth without much valuable insight in the discussion.
Q: Are the graph and observational confidence scores meaningfully comparable?
We agree that normalization alone does not establish comparability. Both pathways assess the same candidate arguments as the QBAF propagates model-assigned base scores through the graph, while the observational judge independently assesses the discussion without receiving the numerical QBAF scores.
To provide empirical support for comparable score interpretation over the populated score ranges, we conducted an additional evaluation across the main benchmark suite to examine whether similar winner scores from the two pathways correspond to similar observed correctness rates. For each pathway, we binned the normalized score assigned to its selected winner and measured the accuracy within each bin. Across corresponding bins containing at least 30 instances per pathway, the mean absolute difference in observed accuracy was 1.4 percentage points, making the two scores closely aligned over the evaluated bins.