acceptodds
Under review as a conference paper at ICLR 2027

ARGORA: Orchestrated Argumentation with Counterfactual Diagnostics for Multi-LLM Consensus

Abstract

Existing multi-expert LLM systems gather diverse perspectives but resolve them through simple aggregation or voting, obscuring which arguments drove the final decision. We introduce , a framework that compiles multi-expert discussions into explicit graphs of supporting and attacking arguments. By formalizing the deterministic aggregation procedure over these graphs as a structural model that admits edge-local interventions, ARGORA can systematically remove individual arguments and recompute outcomes, identifying which argument chains were decisive and whether decisions would change under targeted modifications. We further introduce a correction mechanism that aligns the framework's internal consensus with external judgments when they disagree. Across diverse benchmarks and an open-ended use case, ARGORA achieves competitive accuracy and demonstrates corrective behavior: when experts initially disagree, the framework resolves arguments toward correct answers more often than it introduces new errors. Our implementation is publicly available at https://anonymous.4open.science/r/ARGORA-iclr.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
argumentation graph, multi-llm discussion, counterfactual intervention, consensus generation, mechanistic interpretability, explainability, QBAF

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_LnUmOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tested a multi-agent system where several AI agents discuss a question, compare their reasons and reach a decision using a graph. The paper explores the problem that if using multiple reasoning by different AIs, with structured argumentation graphs, will this be helpful to identify valid claims and settle disagreements. Its main idea is to record which arguments support or challenge other arguments, then test how removing a connection changes the decision.

Critical strengths
  1. Because the graph participates in decision computation, interventions can test their dependencies within the actual aggregation procedure, which naturally embeds the system with some explanations.

  2. The interaction between different AIs itself is an interesting topic to explore.

Critical weaknesses
  1. Why should the additional AI judge be trusted over the argument graph? When the two judgments disagree, what evidence shows that the additional judge is more likely to be correct? This matters because agreement with another AI is not necessarily agreement with reality.

  2. Faithfulness to the graph’s computation does not establish faithfulness to the underlying LLM’s internal reasoning. A valuable follow-up would compare graph interventions with interventions on agent inputs and subsequent generations. A better-designed qualification for causal claims is in need.

  3. Tree shaped argumentation and single edge interventions simplify analysis, interacting or redundant reasons may require joint interventions. A reason can matter collectively while showing little effect when removed individually.

Questions for authors

Are the two confidence scores meaningfully comparable? The correction rule compares confidence from graph calculations with confidence assigned by the AI judge. Both are normalised, but they come from different processes. arxiv.org For example, a graph score of 0.7 and a judge score of 0.8 may not represent the same kind of certainty. What evidence shows that a higher judge score means its answer is more reliable? Normalising numbers so they sum to one does not, by itself, make them equally trustworthy.

1 comment

We thank the reviewer for their thorough review. We address each of the weaknesses and the questions below.

W1: Why should the additional judge be trusted over the argument graph?

It should not be trusted unconditionally, and ARGORA does not do so. Instead, we treat the observational judge as a fallback assessment, in the case where we believe that the argument graph’s decision is too fragile (or more precisely, override when a single edge perturbation already changes the graph’s own winner). This override additionally requires satisfaction of the winner-confidence gate, and improvement in the alignment–perturbation objective relative to leaving the argument graph unchanged (Definition 4.6). As for the performance of the judge itself, since it is also reliant on the ARGORA’s generated transcripts (without the argumentation graph), our evaluations have found it to have comparable performance to ARGORA as shown in Appendix G.9, which makes it a useful corrective signal.

W2: graph faithfulness vs the LLMs' internal reasoning

We actually considered this proposal while designing ARGORA. The main obstacle is computation / token cost, where a direct input intervention requires regenerating every downstream review, rebuttal and base score for each perturbed argument. This cost multiplies over every argument in every argumentation graph, so this work is restricted to fixed-graph interventions.

Our causal claims are scoped accordingly in the paper. The causal object is the aggregation procedure (Prop. 4.1): edge-local counterfactuals state exactly how the decision depends on each argumentative channel, not why an LLM wrote an argument or how it reasoned internally. The design is motivated by the unfaithfulness of free-form reasoning traces, where instead of trusting a trace to reflect latent reasoning, we make the decision depend on an explicit, recomputable object. We will clarify the scope based on our design further in our paper, and we do believe that building a causal system on the underlying LLM’s reasoning is out of scope of our work and leave it as a promising future research direction.

W3: Tree topology and single-edge interventions may miss joint reasoning.

We do agree that single-edge deletions do not capture every way reasons combine. Effects in a QBAF are not additive (App. E.2), so a reason with a small individual effect may matter jointly with others. That said, joint interventions are well-defined in our framework (Lemma C.12, Def. 4.6), where a joint intervention is a set of single-edge deletions applied together. Our reported diagnostics apply one deletion at a time.

Other than that, the tree topology is a deliberate consequence of ARGORA’s discussion protocol, detailed in Appendix D. Each main argument anchors a separate discussion, and each supplementary argument therefore has an explicitly designated parent. We’d like to also clarify that redundancy is handled at construction, using contextual orthogonality pruning (Sec. 4.2, App. D.4): near-duplicate siblings are removed before evaluation so that single deletions stay informative. We went ahead with this design choice as we found that unstructured and unguided discussion processes can lead to endless spirals of circular reasoning or experts going back and forth without much valuable insight in the discussion.

Q: Are the graph and observational confidence scores meaningfully comparable?

We agree that normalization alone does not establish comparability. Both pathways assess the same candidate arguments as the QBAF propagates model-assigned base scores through the graph, while the observational judge independently assesses the discussion without receiving the numerical QBAF scores.

To provide empirical support for comparable score interpretation over the populated score ranges, we conducted an additional evaluation across the main benchmark suite to examine whether similar winner scores from the two pathways correspond to similar observed correctness rates. For each pathway, we binned the normalized score assigned to its selected winner and measured the accuracy within each bin. Across corresponding bins containing at least 30 instances per pathway, the mean absolute difference in observed accuracy was 1.4 percentage points, making the two scores closely aligned over the evaluated bins.

Reviewer_sSrzOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The authors propose ARGORA, which is a framework that allows multiple AI models to collaborate on a question and represents their arguments as a graph consisting of supporting and opposing arguments. By making targeted changes to this graph, it determines which arguments are decisive for the decision. An additional evaluation can adjust the result.

Critical strengths

The paper combines multi-LLM debate with argumentation formalization and counterfactual analysis as an evaluation step, which is a very interesting approach. In tests, ARGORA achieves good results on several tasks compared to simple voting and debate procedures.

Critical weaknesses

The whole framework relies on the ability of LLMs to build the argumentation graph and all relations correctly. The question remains, how well this step is working. A deeper analysis, comparing the LLM-constructed argumentation graphs agains expert opinions would be beneficial.

Questions for authors
  • How do you demonstrate that the argumentation graphs adequately capture the relevant reasons?
  • How is the quality of the argumentation graphs validated?
  • How independent is the LLM-as-a-judge model that is used as the external override signal?
  • How much does ARGORA's additional inference cost compared to majority voting and other debate procedures? Is the gain in accuracy significant enough to justify these costs?
1 comment

We thank the reviewer for the thorough review. The concern about the reliability of the argumentation graph construction in the critical weakness is directly addressed by ARGORA’s design choice of using a structured discussion protocol that allows for a controlled argument and relation extraction (as detailed in Appendix D), and structural integrity holds by construction. We further clarify with the four questions below.

Q1. How do you demonstrate that the argumentation graphs adequately capture the relevant reasons?

The participating experts in ARGORA generate their reasoning in the context of the discussion. In detail, the discussion topic, extracted main task, and key elements from the orchestrator provide shared context throughout the discussion process (Appendix D.1). Each supplementary argument is generated in response to an explicitly identified target as specified in Algorithms 3–5 to avoid inferring argumentative relationships from an unstructured transcript. We chose this design because unguided discussions tended toward circular reasoning with little new insight. In the end, we explicitly verified the correspondence between the generated discussions and their argumentation graphs through manual inspection and automated sanity checks. Since each of the retained arguments and relations generated by the experts can directly be mapped one to one as our argumentation graph, this design allows us to guarantee that the argumentation graph reconstructs the discussion faithfully.

Q2. How is the quality of the argumentation graphs validated?

This is an extension to our answer to the previous question: during the pilot tests of our implementation, we directly compared the model-generated discussion transcripts with the resulting graphs to check whether the arguments and relations are valid and consistent.

To be more specific, we developed a sanity-check script to verify that the graph faithfully reconstructs the recorded discussion. Since ARGORA elicits structured responses with explicit stances and identifiable targets, the relevant fields can be extracted through deterministic parsing (extraction using regex for example) and allows for an automated comparison between the recorded outputs and the graph representation without requiring the use of another LLM or some other heuristic methods. When we ran the script over the test set containing 4500 different graphs over 1500 generated discussion logs (each discussion was limited to 3 main arguments here), we measured a 100% match for the main, supplementary arguments and the relations. That being said, this does not assess the quality of the arguments themselves, and a comparison against human expert annotations is valuable future work.

Q3. How independent is the LLM-as-a-judge model that is used as the external override signal?

The default LLM-as-a-judge reuses the same backbone as the experts and Orchestrator, but its judgement is completely independent of quantitative QBAF evaluation. The judge only receives the argument statements and a compressed transcript without the numerical QBAF scores or the winning main argument as calculated by ARGORA. So in essence, the judge is receiving information that would only be accessible through the discussion process as we describe in Appendix D. Our table in Appendix G.9 also compares LLM-as-a-judge and ARGORA’s default QBAF decisions where they disagree.

Q4. How much does ARGORA’s additional inference cost compared to majority voting and other debate procedures?

This is answered in Appendix G.12. The additional inference cost varies for each of the benchmark based on the difficulty, but on the hardest benchmark from our suite (GPQA-Diamond), it is approximately equal to majority voting that uses 18 samples, or ~6x MV3, ~4.5x MAD, and ~2x ReConcile (using their default settings). We’ve additionally tried raising MAD/ReConcile round counts to match or exceed our token budget, with no accuracy gain. Our evaluations against majority voting as shown in Table 22 also show the same phenomenon.

Method (5 runs)RoundsAccuracy (mean ± SD)Output tokens / instance
MAD1339.39 ± 1.97≈11,700
ReConcile1640.40 ± 1.24≈10,500
ARGORA–44.65 ± 1.50≈8,900

More importantly, ARGORA's contribution extends beyond gain in accuracy. The additional inference cost also supports counterfactual decision analysis and auditability, which are capabilities that accuracy alone does not capture. Therefore, ARGORA is not positioned as a cost-efficient replacement for majority voting in large-scale inference workloads. Rather, ARGORA is intended for applications where the quality and inspectability of individual decisions may justify additional computational costs, particularly high-stakes analytical settings such as the cybersecurity use case scenario presented in our paper (in Section 6).

Reviewer_8wcYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

ARGORA is a scaffold, in which one backbone LLM is given several "expert" personas. When ARGORA is used to answer a question, each persona proposes an answer, then writes arguments for and against each answer in three rounds like a review & rebuttal process, which expands each original answer into an argumentation tree. A coordinator implements a set of deterministic rules with LLM judges and calculates a score for each original answer, which determines the winner. The tree structure allows us to locate which argumentation branch led to the winning answer.

With GPT-4o-mini with backbone, on severn benchmarks, ARGORA beats direct prompting and 3-vote majority on most of them. On GPQA diamond, ARGORA beats existing similar scaffolds. Compared to direct prompting, ARGORA costs 12x output tokens and 9x wall clock time, comparable to 18 majority voting.

Critical strengths

None

Critical weaknesses
  • The lead of the proposed method, ARGORA, over baseline is quite small on GPT-4o-mini, and even smaller on GPT-5-mini. And the selected baselines don't seem particularly strong: MAD and RECONCILE are from 2024. The limited benchmarks, backbone model selection, and modest margin over old baselines---it doesn't give me a lot of confidence in the empirical efficacy of the proposed method, despite the significant compute overhead (18x majority vote).
  • Some baseline results look off. Majority vote sometimes underperforms direct prompting (TruthfulQA 0.792 vs 0.758), which suggests issues with tie-breaking or something else. Many headline differences are smaller than single run standard deviation.
  • Appendix G.9: when the judge and the tree disagree, the judge is right 51 times and the tree 31 times (107 vs 35 with a GPT-5-mini judge). Using the judge directly would likely score higher than the tree, so the argument for building the tree is mostly for "interpretability", not accuracy.
  • Scaffold design shouldn't optimize hyperparameters on tasks it's tested on. However, the “best semantics” column and the override settings are chosen from a grid evaluated on the same benchmarks as the test set.
Questions for authors
  • Can you report the judge’s accuracy on all benchmarks, as well as a majority vote over the experts’ main argument labels?
2 comments

We acknowledge the concerns and remarks as presented by the reviewer, and would like to directly address all of the weaknesses and questions as follows.

W1: Performance and baseline selections

We respectfully disagree with the statement that publication date makes a baseline weak; MAD and ReConcile remain the canonical works in this field and remain relevant in recent literature. As for the claim on a modest margin, across five full runs on GPQA Diamond, ARGORA leads MAD and ReConcile with non-overlapping 95% CIs (Table 2; Welch’s t-test, p<0.002 for both), and we additionally ran a new experiment raising their round counts which also resulted in no accuracy gain (same applies to majority voting as shown in Table 22).

Method (5 runs)RoundsAccuracy (mean ± SD)Output tokens / instance
MAD1339.39 ± 1.97≈11,700
ReConcile1640.40 ± 1.24≈10,500
ARGORA–44.65 ± 1.50≈8,900

We also compare with ArgLLM (AAAI 2025) on its own benchmarks as shown in Table 13 in Appendix G.7. Finally, we clarify that the “18x majority vote” counts single-model samples (Table 23), equivalent to 6x the MV3 baseline of Table 1.

W2: Baseline results

The large TruthfulQA gap did look off, so we rechecked the scripts and logs for every benchmark. We confirmed a parse failure affecting only TruthfulQA on the majority vote baseline; the corrected majority-vote accuracy is 0.800 (we will revise the paper). All other results are correct as reported, including the three MuSR tasks where majority voting slightly underperforms. We also confirm that tie-breaking is not the cause as ties occur in only 0-1 instances per dataset.

We would nonetheless like to emphasize that there is no general guarantee that majority voting improves accuracy as documented in recent LLM research such as Baek et al. (D-FUSEr, ICML 2026) which shows concentrated errors degrade majority-vote aggregation. Regarding the statement that many improvements are smaller than a single-run standard deviation, we again point to our five-run GPQA-Diamond from Table 2 (where we explicitly chose GPQA-Diamond as it is the most difficult benchmark from our suite).

W3: Judge performance

To answer this, we present the table to report judge-only accuracy and a vote over main-argument labels for the Table 1 run as asked by the reviewer.

BenchmarkARGORAJudge onlyVote over main arg labels
MMLU-Pro0.6360.6220.602
MedQA0.8160.8200.796
TruthfulQA0.8820.8700.864
GPQA Diamond0.4550.4600.424
MuSR (MM)0.6640.6560.636
MuSR (OP)0.5230.5000.504
MuSR (TA)0.5640.5520.556

Our experiment in App. G.9 tests whether the judge acts as a valid second signal for the override about as accurate as the graph without repeating its errors. This observed advantage of 51 vs 31 instances is a small absolute net 20 of 2,454 instances (+0.8pp). In the run above, the judge is instead lower than ARGORA on 5 of 7 tasks, so the sign changes between runs and we claim neither aggregator is always more accurate. The wider 107 vs 35 gap pairs a GPT-5-mini judge with a GPT-4o-mini discussion and is to be expected from a stronger model and not a better aggregator. We also point to Table 12 results which show ARGORA always beats per-expert standalone performance.

We agree that the graph’s main value is inspectability, a main contribution of our paper. Adopting the judge’s answer directly would be an alternative aggregation policy, but would not supply the computed dependency tests or the record of consensus revision. ARGORA thus combines competitive accuracy with an aggregation procedure that can be explicitly inspected and tested.

W4: Hyperparameter selection

No comparison uses per-benchmark selection. All comparisons against baselines use DF-QuAD on every benchmark; we chose this as the default before evaluation, following prior related work by Freedman et al. (ArgLLM, AAAI ‘25), and will state this in the revision. The “Best” column only reports sensitivity to the choice of semantics.

The override configuration was also fixed before the main evaluation, on a 100-sample MMLU-Pro pilot drawn with a different seed from the Table 1 split with no overlap. It was then applied unchanged to all seven benchmarks, six of which played no part in the selection. The sweep in App. G.10 is a post-hoc sensitivity analysis and the current wording in the paper wrongly suggests the default hyperparameter was selected from it, and we will correct this. The choice also matters little in that the pooled result moves by at most 3 of 2,454 instances (Table 17) across the full sweep grid, and all headline comparisons hold pre-override (Tables 1 and 2).

Thank you for the response. Some of my concerns are addressed, but some remain.

W1. The GPQA results seem valid. However:

  • The strongest baseline is not included in the 5 runs comparison. In Table 1, “ARGORA-like (modified CoT) MV3” scores 0.459 (against 0.455 for ARGORA) on GPQA Diamond. In other words, ARGORA beats the external baselines but not a plain 3-vote baseline using the same prompt.
  • My critique about GPT-5-mini results is ignored.
  • The compute-matched majority-vote sweep is weak.
    • Table 22 uses direct prompting, not the stronger modified-CoT prompt
    • k only up to 11, while ARGORA’s budget is about 18 samples
    • only 100-question subsets of only four datasets
    • The curve is flat from k=2, which suggests low sample diversity rather than saturated voting. I don't think that the conclusion about voting being “bounded by the accuracy of that backbone regardless of how many samples are drawn.” is empirically supported.

W2 I'd glad you fixed the TruthfulQA bug. My critique was about differences smaller than run-to-run variance. Your response cites the 5-run GPQA comparison, which, as above, leaves out the strongest baseline. Looking at Table 1, with N=198–500 and a single run, only TruthfulQA and maybe Murder Mystery are clearly outside noise. The new Murder Mystery 5-run std deviation is about the size of most of these gaps.

W3 (Judge vs. tree) Based on the new results, the tree seems ahead by +0.85pp. In G.9, the judge was ahead by +0.8pp. Across the two runs, the tree and a judge reading the same discussion are tied within about ±0.8pp.

Importantly, the new judge-only baseline beats every Table 1 baseline on most tasks. So the accuracy gain comes from generating the multi-persona discussion, not from the argumentation graph and its scoring. This shifts the paper’s contribution from performance to interpretability, which the experiments don’t really evaluate. The new “vote over main-arg labels” column (no discussion) is mostly below ARGORA, except on TruthfulQA (0.864). That suggests persona diversity alone explains much of the TruthfulQA gain.

W4: Hyperparameters DF-QuAD was the fixed default for every baseline comparison, and the override settings barely change the pooled results. That addresses my concern about tuning on the test set.

Overall, the response fixes several real issues. But my concerns about the method’s effectiveness against strong, compute-matched baselines, and about what the argumentation graph adds over the discussion itself, still hold. I’m raising my rating to 2 and still recommend rejection.

Comments on the paper

1 comment

Summary of revisions

We thank all reviewers for their feedback. We have uploaded a revised manuscript with the changes marked in blue, and a summarized table of changes is provided below for reference.

RevisionLocationAddresses
Clarified the origin of the override configuration (CstateC_{\text{state}}, λ=0.05\lambda = 0.05, τ=0\tau = 0) before the main evaluation.App. G.10 (p. 48, 50)Reviewer 1, W4
Stated that DF-QuAD was fixed as the default semantics before evaluation.Sec. 5.3 (p. 7)Reviewer 1, W4
Corrected the TruthfulQA majority-vote (MV3) accuracy from 0.758 to 0.800 after fixing a parsing error.Table 1 (p. 7)Reviewer 1, W2
Revised the framing on the judge-vs-graph discussion for clarification.App. G.9 (p. 45)Reviewer 1, W3; Reviewer 3, W1
Added Table 24 with MAD and ReConcile at raised round counts for token-matched comparison.App. G.12 (p. 53)Reviewer 1, W1; Reviewer 2, Q4
Clarified the scope of the counterfactual explanations.App. K (p. 63)Reviewer 3, W2

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.