acceptodds
Under review as a conference paper at ICLR 2027

Consensus Is Not Evidence: Evidence Lineage for Replication-Safe Multi-Agent Reasoning

Abstract

Many multi-agent LLM systems treat repeated agreement as additional support. When reports share upstream evidence, this can count one observation multiple times and suppress a minority report grounded in new evidence. We formalize evidence lineage by grouping traces by answer-relevant upstream observations and show that marginal accuracy and pairwise correlation cannot identify consensus reliability. Even identical answers and calibrated reliabilities can require opposite Bayes decisions depending on whether reports are independent or replicated. This motivates replication invariance: adding a non-informative replica should change neither prediction nor confidence. RIDE operationalizes these principles through an acquire–group–aggregate protocol that audits lineage structure, acquires distinct upstream evidence when needed, collapses exact replicas, and aggregates within and across lineages without adding mass to duplicates. The framework separates failures to obtain new evidence from failures to preserve its influence during aggregation. Controlled interventions isolate answer-bearing support as the driver of acquisition and show that handling must preserve an informative lineage against redundant votes. In a non-oracle study with parameters fixed before evaluation, RIDE-CAL improves over the best fixed source by 7.5 points and reduces binary answer NLL from 1.646 to 0.650. Across 3,600 exact-replication attacks, its replication-invariant design yields a 0% prediction-or-confidence change rate versus 69.08% for confidence weighting. Replication safety persists under provenance corruption, while false merges and aggressive overlap grouping emerge as accuracy risks. Reliable multi-agent scaling should therefore optimize and audit evidence lineages, not raw response count.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multi-agent LLM systems, LLM agents, provenance, calibration, self-consistency, retrieval-augmented generation, source reliability, Bayesian decision theory

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8E9bOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that multi-agent LLM systems overcount agreement when several reports share upstream evidence. It formalizes this with evidence lineages, proves that correlation and calibrated reliabilities cannot identify the right aggregate without dependence information, and characterizes aggregators that ignore added replicas. The proposed method, RIDE-CAL, removes duplicate traces, groups the rest by evidence fingerprint, and combines lineages with a calibrated reliability model. On 200 HotpotQA tasks with three retrievers, it reaches 52.0% accuracy against 50.5% for confidence-weighted voting, lowers answer NLL from 1.646 to 0.650, and never changes its output under 3,600 exact-replication attacks.

Critical strengths
  1. The paper studies a practical. Votes that share upstream evidence are counted as independent, so a minority with new evidence loses (Appendix D.3).

  2. The evaluation protocol is disciplined. Parameters are fixed on development data before the confirmation run. The paper reports paired tests and states that its accuracy gain is not significant.

  3. The provenance-corruption study shows that false merges hurt accuracy while false splits do not, which is a practical rule for building lineage labels.

Critical weaknesses

1. My main concern is that the 0% replication result holds by construction and misses the motivating case.

A replica is a copy made "without a new model call" (line 208), so Theorem 3 does not cover the motivating example of three agents that read the same context (line 42). Algorithm 2 also deletes records with a repeated trace identifier (line 816). If the injected copies keep their identifiers, any method with this step scores 0%, yet the paper does not report the audit for its own exact-replica collapse baseline. The relevant test uses near-replicas, such as new samples from the same context or copies with new identifiers.

2. The non-oracle study has no replication imbalance to correct.

Each retriever contributes exactly two traces (line 323), and exact-replica collapse matches instance majority on every metric (Table 5). Against confidence-weighted voting, RIDE-CAL gains 1.5 points (95% CI −0.5 to 4.0), and without document provenance it falls to 48.5% (line 430), below instance majority.

3. The calibration gain appears where all sources agree, so lineage does not explain it.

It comes from the 85 tasks without source conflict (NLL 2.991 to 0.637, line 400), while on the 115 conflict tasks RIDE-CAL is no better than confidence weighting (0.659 vs. 0.651). RIDE-CAL fits two Platt maps on development data (lines 785–786), and the paper describes no such calibration for confidence weighting, so that baseline needs the same calibration.

4. The controlled studies rely on an oracle.

Adding the paragraph that the design defines as the only answer-bearing evidence raises accuracy by 34 to 43 points (Table 1), which is expected. The handling contrasts compare voting with a rule that routes to the source known to hold the gold support (line 260), and without that oracle, one shared lineage against one independent lineage is a tie.

5. The theory is mostly known, and the paper compares with no dependence-aware method.

Theorem 1 is the correlated Condorcet result (Ladha, 1992), Theorem 2 is standard double counting, Theorem 3 follows from the definition, and Theorem 4 specializes Ao et al. (2026) by the paper's own account (line 242). No experiment estimates the acquisition and handling terms. The paper cites source-copying detection (Dong et al., 2009) and CAGE-CAL (Huang et al., 2026) but uses neither as a baseline, and it omits the surprisingly popular algorithm (Prelec et al., 2017), which targets the same failure of a correct minority.

6. The evidence is narrow for claims about multi-agent reasoning.

The non-oracle results come from one 200-task HotpotQA set with retrieval ensembles of a single, unnamed Qwen model, and agent communication such as debate appears only in the oracle studies. Lineages are formed by hashing the rendered context (line 817), not by the answer-relevant evidence of Definition 1.

Questions for authors
  1. Do the injected copies keep the trace identifier that Algorithm 2 removes, and what do the exact-replica collapse baseline and RIDE-CAL show when the added trace is a new sample from the same context?

  2. Can the authors run a non-oracle experiment with unequal replication, for example six traces from one retriever and one from each of the others?

  3. What NLL do instance majority and confidence weighting reach after the same development-fitted Platt map?

  4. What accuracy does RIDE-CAL reach in the controlled setting without the designated-source rule?

  5. How does RIDE-CAL compare with source-copying detection (Dong et al., 2009) or the surprisingly popular algorithm (Prelec et al., 2017)?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 8E9b

We sincerely thank the reviewer for the time and care invested in this detailed, technically specific review, and for the constructive comments that have helped us improve both the experiments and the presentation. We address each weakness and question below, with the corresponding tables.

W1. The 0% replication result and the motivating case

The paper reports the exact-replica result as a construction check: Step 1 of Algorithm 1 removes repeated trace identifiers, so an exact replica cannot change the output (Sec. 5.5). The case that matters — near replicas with fresh trace identifiers — is measured directly in Sec. 5.5.

Table 1. Same-evidence resampling on 200 tasks (paper Sec. 5.5)

SourceRandom-subset agreementImplied modal-answer change
Lexical (BM25)0.73526.5%
Dense0.70929.1%
Complementary PRF0.70729.3%

Two independent trace subsets are drawn from the same per-source contexts, and the modal answers of the two subsets are compared. Agreement of 0.735 means that on 26.5% of panels a fresh sample from the same evidence changes the modal answer. The table therefore quantifies the stochastic drift that exact-replica invariance does not cover, and it is the empirical counterpart of the reviewer's near-replica case.

Table 2. Count sensitivity of decision mass (paper Sec. 5.5)

MethodVoting-share slope with respect to trace count
Instance majority1.000
Confidence-weighted instances0.919
Hard lineage voting0.000 (by construction)
RIDE-CAL−0.0007

The slope measures how much extra decision weight a method gives a lineage as its trace count grows; zero means the method is count-invariant. Instance-level methods accumulate weight linearly with replication, whereas RIDE-CAL's lineage weight does not grow with the number of resamplings. Table 8 in the appendix also reports the replica-collapse baseline (35.25% accuracy, NLL 0.607); on the frozen bank it coincides with flat aggregation. Appendix E reports the controlled replication attacks: instance voting changes in 58.5% of 1,600 attacks, RIDE-CONTROLLED in 0%.

W2. The non-oracle study and replication imbalance

This is now the main non-oracle experiment. Sec. 5.2 reports six unequal-replication layouts in two independent confirmation batches.

Table 3. Response-count imbalance confirmation (paper Table 2, Sec. 5.2)

BatchLayoutRIDE correctInstance correctConfidence correctRIDE wrongYouden's J
A6:2:247.928.139.617.630.3
A2:6:245.130.543.914.330.8
A2:2:652.232.642.415.037.2
B6:1:159.822.444.922.237.6
B1:6:162.929.953.624.838.1
B1:1:660.630.844.922.837.7

Layouts are a:b:c = numbers of sampled traces on the lexical, dense, and complementary-PRF retrievers. Correct is the share of eligible tasks where the method selects the unique smallest post-merge lineage containing the gold answer; wrong is the share where it selects the unique smallest lineage carrying a wrong answer. Batch A and Batch B are independent confirmation batches with different frozen task pools and balanced accuracies of 35.1% and 39.6%; non-inferiority is computed within each batch. Across all six layouts RIDE-CAL selects the correct minority lineage in 45–63% of cases versus 40–54% (confidence) and 22–33% (unweighted), with accuracy within 2 points of the batch baseline.

W3. Calibration by conflict status and matched calibration

The comparison has been rebuilt. Calibration is stratified by whether post-merge lineage representatives disagree, and every method's probabilities use development-fitted Platt maps.

Table 4. Calibration by post-merge conflict status (paper Table 3, Sec. 5.3)

StratumMethodnAccuracy (%)NLLBrierECE-10
ConflictCalibrated flat2,22323.60.5510.1830.091
ConflictRIDE-CAL2,22324.10.5480.1810.086
Non-conflictCalibrated flat1,52752.20.6890.2470.030
Non-conflictRIDE-CAL1,52752.20.6840.2450.039

Conflict panels are those where post-merge lineage representatives disagree; non-conflict panels are those where they agree. All maps are fitted on development data only. The differences between the two methods are small in both strata, and the paper presents calibration as a supporting stability result. The baselines in Table 6 also use development-fitted maps, as its caption states.

W4. The controlled studies and the oracle setting

The controlled setting is described as "an oracle setting where passage roles are known to the experimenter" (Sec. 3.4), and Sec. 5.6 compares rules on "identical controlled trace multisets" to isolate handling. The controlled study serves as a mechanism diagnostic, while the deployable evidence comes from the non-oracle full evaluation (Sec. 5.4) and the imbalance confirmations (Sec. 5.2). As a source-free check, removing source-identity features from the selector and refitting on development leaves accuracy essentially unchanged (35.28% versus 35.47%; Appendix D.3, Table 8).

W5. Theoretical positioning and dependence-aware baselines

We agree with the characterization: Theorem 1 restates the correlated Condorcet result (Ladha, 1992); Theorem 2 is the double-counting argument for shared evidence; Theorem 3 is a characterization that follows from the definition of replica equivalence; Theorem 4 specializes the decomposition of Ao et al. (2026). The paper presents these results as the framework that fixes the object of study and yields testable predictions, and its claims rest on the empirical mechanism study.

Two additions respond to this weakness. First, Sec. 5.5 reports empirical decomposition under 0/1 task loss relative to a three-source reference: acquisition gaps of 0.042 (one source) and 0.010 (two sources), and a handling gap of −0.003; the paper labels this as a proxy estimate. Second, Appendix D.4 adds dependence-aware baselines.

Table 5. Acquisition and handling gaps (paper Sec. 5.5)

QuantityOne sourceTwo sourcesThree sources
Acquisition gap (0/1 task loss)0.0420.0100
Handling gap (0/1 task loss)––−0.003

The acquisition gap measures the loss attributable to evidence that was not acquired, holding the aggregation rule fixed: it shrinks as more sources are observed and reaches zero with all three. The handling gap is the residual loss after all acquired traces are in the transcript and is close to zero, which is the empirical analogue of the acquisition–handling decomposition in Theorem 4. This is a proxy estimate under 0/1 task loss, as Sec. 5.5 states.

Table 6. Dependence-aware baselines (paper Table 10, Appendix D.4)

MethodAccuracy (%)NLLBrierECE-10Additional calls
Source-copy33.090.5780.1960.0210
CAGE-CAL33.810.5930.2030.0230
Surprisingly-popular (Prelec)26.130.4970.1660.0280
RIDE-CAL35.470.6040.2070.0650

All rows use the same 22,500-trace pool, so no additional calls are needed. NLL and Brier use each method's development-fitted maps. RIDE-CAL optimizes for replication invariance and minority-evidence recovery, not for NLL minimization on balanced panels. The three dependence-aware baselines, after their own calibration, achieve lower NLL but substantially lower accuracy (−1.7 to −9.3 points). This reflects a design choice: RIDE-CAL trades calibration tightness on consensus tasks for correctness on minority-evidence tasks (Sec. 5.2). On the conflict stratum (Table 3), where lineage matters most, RIDE-CAL attains NLL 0.548 versus 0.551 for calibrated flat.

W6. Scope of the evidence, lineage resolution, and debate

The evaluation covers three datasets and five models (3,750 test units, 22,500 traces), with per-cell results in Table 4 and the model roster in Appendix A. Lineage resolution is tested directly (Sec. 5.4): fingerprint, document and sentence-level lineages give 35.5% and 35.7% accuracy, so the qualitative result does not hinge on one granularity. Lineage is operationalized by hashing the rendered context, which approximates the answer-relevant evidence state of Definition 1; Sec. 5.4 and Appendix D.6 report how this proxy behaves under false merges, false splits and overlap grouping. Debate appears in the controlled study (Appendix E), and Sec. 6 states the scope of that evidence.

Questions

  1. Identifiers and near replicas. Exact replicas keep their identifiers and are removed by Step 1; the paper reports this as a construction check. For new samples from the same context, Table 1 reports a 26–29% modal-answer change while Table 2 shows that RIDE-CAL's voting share stays count-invariant (slope −0.0007). Appendix E reports the controlled replication attacks.
  2. Non-oracle unequal replication. Six layouts in two batches (Table 3).
  3. NLL after the same development-fitted Platt map. Instance majority and confidence weighting both reach 0.607; RIDE-CAL reaches 0.604 (Appendix D.3, Table 8).
  4. Controlled setting without the designated-source rule. Without privileged roles the controlled rule reduces to no-lineage voting; Appendix E reports that condition and notes that its contrast with independent-only is a handling diagnostic. In the non-oracle setting, removing source-identity features gives 35.28% versus 35.47%.
  5. Dependence-aware comparison. Source-copy and surprisingly-popular baselines are reported in Table 6 with the caveats above.

Summary

The paper's contributions are the measurement of source-level versus lineage-level replication (48.0% of panels merge at least two nominal sources), the near-replica and count-invariance analysis (Tables 1–2), and the minority-recovery mechanism under unequal replication (Table 3), with the merged-lineage boundary discussed in Sec. 7.

We thank the reviewer again for the careful and technically specific reading; the points raised have directly shaped the revision, and we would be glad to address any remaining concerns in the discussion.

Reviewer_vxoROpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses a failure mode in multi-agent systems where agents that share the same incomplete evidence agree on a wrong answer and outvote a lone agent whose new evidence leads it to the correct one. The paper builds a theory around why consensus requires provenance through a series of mathematical proofs. It then introduces the main contribution of the paper RIDE which is a framework for aggregating evidence and reweighting answer confidences based on this aggregation. The authors then empirically prove the effectiveness of their method through a small empirical study with two datasets.

Critical strengths

The idea of taking into account the lineage of the evidence to make a decision is novel and does show ingenuity on the author's part. Furthermore, they build a system that does not double count evidence towards the voting which is a novel contribution towards building better multi-agent systems

Critical weaknesses

The main critical weakness is the extremely poor preparation of the empirical experiment. The experiment does not specify important key factors like the prompt, model selection, model generation parameters, how they served the models and more nor justifies decisions like dataset choices, dataset subselection, overall experiment design, metrics and more. This lack of detail leads to a lack of understanding of the conclusions of the empirical experiments and a lack of replication of the experiments in the future. The authors also do not test alternative methods to multi-agent protocols like “Retrieval-Augmented Generation with Conflicting Evidence” (Wang et al 2025), “Diverse Evidence, Better Forecasts: Multi-Agent Deliberation Under Information Asymmetry” (Li et al 2026), “Beyond Memory Majority: Latent-Source Reasoning for Multi-Agent Memory Arbitration” (Lin et al 2026) and many others. Finally, many hyperparameters in the retriever do not have an ablation experiment detailing their effect on downstream performance.

The second critical weakness is the poor preparation of the manuscript itself. While CS is a technical field with its own share of jargon, this paper does not contextualize their use of jargon like oracle/non-oracle, document-regime, output cap, complementary PRF, etc. Furthermore, this use of not-defined jargon leads to sets of extremely dense and hard to parse sentences like “Complementary PRF is designed to prevent construction-time identity without assuming semantic independence.” This leads to a lack of understanding of the paper and its main thesis and results.

The paper appears to be almost 100% AI written and is consequently almost impossible to read and failse to develop its contribution and demonstrate how this compares to related work, placing the contribution in the framework of the literature.

Questions for authors
  1. Does this paper relate to work in multiagent debate where agents revise their answers over multiple rounds?
  2. How would this technique be used and when, by whom?
  3. What is the relevance of the theorems to the technique? It is impossible to follow.
2 comments
AuthorsOpenReview ↗

Response to Reviewer vxoR

We sincerely thank the reviewer for the time and care invested in reading our submission, and for the constructive comments. They have directly shaped the revision: the experimental reporting is now complete, the related-work positioning is explicit, the retriever ablation is included, and the manuscript has been rewritten for readability. We answer each point below with the corresponding tables and would be glad to clarify any remaining concerns.

W1. Incomplete reporting of the empirical setup

The revision specifies the elements the reviewer listed:

  • Models and allocation: five models at 250 test questions per dataset-model cell (3,750 task-model units; Appendix A, Table 5).
  • Generation: temperature 1.0, 500-token output cap, 6,000-token input budget, three retries; the prompt is reproduced verbatim (Appendix A).
  • Retrieval: BM25 (k1 = 1.2, b = 0.75), normalized all-MiniLM-L6-v2 dense encoder, four documents per source, complementary PRF with eight feedback terms; retrieval, task selection and trace manifests are frozen before generation.
  • Data and splits: three multi-hop QA datasets, a stratified development/test split, and no test answer used to fit a selector or calibration map.
  • Metrics and statistics: normalized exact match, binary NLL, Brier score, 10-bin ECE, paired bootstrap, exact McNemar, Holm correction, with the statistical unit stated (task or task-model unit).
  • Artifacts: task, retrieval, trace, selector and calibration records are recorded and released.

Table 1. Model roster (paper Table 5, Appendix A)

ModelAPI identifierRole in the evaluation
DeepSeek-V3.2deepseek/deepseek-v3.2full evaluation and confirmations
Gemma-3-27B-ITgoogle/gemma-3-27b-itfull evaluation and confirmations
Llama-3.3-70B-Instructmeta-llama/llama-3.3-70b-instructfull evaluation and confirmations
Mistral Large 2407mistralai/mistral-large-2407full evaluation
Qwen3.5-Flashqwen/qwen3.5-flash-02-23full evaluation and confirmations

Each dataset contributes 250 test questions per model; the full evaluation therefore contains 3,750 task-model units and 22,500 traces (two sampled traces per each of three retrieval sources). Development questions and the exact per-cell allocation are listed in Appendix A, Table 5. The independent imbalance confirmations use Llama-4 Maverick in place of Mistral Large 2407, as the table caption in Appendix A states.

W2. Comparison with related multi-agent methods

The three works named by the reviewer (Wang et al., 2025; Li et al., 2026; Lin et al., 2026) operate at a different stage of the pipeline: they redesign upstream retrieval, multi-round deliberation, or latent-source memory arbitration, thereby changing the inputs that agents receive. Our contribution is an aggregation rule over a fixed response panel whose contexts may already overlap. Direct empirical comparison is therefore not applicable—the methods solve different problems and cannot be substituted for one another under the same protocol. We consequently position them only in Related Work (Sec. 2), clarifying this distinction.

W3. Retriever hyperparameters

Appendix D.5 reports two retriever studies. The first is a ten-configuration grid over top-k and PRF terms on two datasets; the second is a four-configuration confirmation on three datasets.

Table 2. Ten-configuration retriever ablation (paper Table 11, Appendix D.5)

ConfigurationCoverageMergeLineagesFlatConfidenceRIDE-CAL
Base50.455.42.4444.246.044.6
Lexical-k251.31.32.9942.043.843.8
Lexical-k654.01.32.9946.448.749.1
Dense-k248.254.02.4644.245.142.9
Dense-k650.455.42.4445.145.545.5
PRF-k2, t451.80.92.9943.345.545.1
PRF-k4, t452.256.32.4345.546.045.1
PRF-k4, t1651.347.32.5243.344.643.8
PRF-k6, t852.70.92.9943.845.545.1
PRF-k6, t1653.10.92.9944.245.146.4

Each configuration is evaluated on 224 task-model units with six traces per unit. Coverage is the share of tasks whose supporting documents are retrieved; merge is the share of panels in which at least two nominal sources return identical contexts and are merged by the fingerprint; lineages is the mean number of post-merge lineages per panel; the last three columns are accuracies (%). The grid shows that RIDE-CAL tracks or exceeds flat aggregation across settings, with gains ranging from +0.4 to +2.7 points, while the merge rate and lineage count change substantially with the retriever configuration.

Table 3. Three-dataset retriever confirmation (paper Table 12, Appendix D.5)

ConfigurationUnitsCoverageMergeFlatRIDE-CALΔ (points)
Base48039.449.431.533.1+1.67
Dense-k648040.649.432.535.0+2.50
Lexical-k648042.90.635.437.5+2.08
PRF-k6, t1648043.10.432.532.9+0.42

This confirmation uses 480 task-model units per configuration drawn from all three datasets, with four models. Δ is RIDE-CAL minus flat accuracy. The table complements Table 3: the same qualitative ordering appears on a larger, three-dataset sample, with the gain depending on how many panels actually merge.

W4. Manuscript preparation, terminology, and readability

This criticism is fair, and we address it head-on. As the AI-use statement in the submission disclosed, we used generative AI tools to assist with translation, language editing, and code generation. In hindsight, the language editing did more harm than good: it produced prose that is dense, overly uniform in tone, and inconsistent about where terms are defined—the very problems the reviewer lists. Therefore, we rewrote large parts of the manuscript:

  • Sentences were split. We substantially shortened and simplified the prose: the abstract and introduction now use short, declarative sentences, and the dense constructions the reviewer quotes are gone; definitions, methods, and claims are stated in separate sentences rather than stacked.
  • Terminology density was reduced. Every term the reviewer lists is now defined where it first appears and collected in the glossary (Appendix B): document regime is the passage-composition cell over which support and distractors are crossed; output cap is the 500-token generation limit; complementary PRF is the deduplicated pseudo-relevance-feedback retriever; evidence fingerprint is the hash of rendered document and sentence IDs; post-merge lineage is the group formed after such hashing.
  • Structure was clarified. The paper now separates theory, method, experimental design, results, and limitations, and the theory section closes with an explicit statement of how the four results shape the method, with Theorem 3 tied to Steps 1–2 of Algorithm 1.

We would be happy to revise further if specific passages remain unclear to the reviewer.

Q1. Relation to multi-round debate

The contribution is at the aggregation stage, not the generation stage. Debate can refine reasoning over passages agents have already retrieved, but it cannot introduce an observation that no agent retrieved; Sec. 6 states this scope. Debate appears in the controlled study (Appendix E, debate/shared conditions), where the evidence state is fixed by design. The paper's claims concern aggregation over a fixed response panel and its interaction with shared context.

Q2. How, when, and by whom the method is used

Sec. 6 gives the deployment guidance: RIDE-CAL is a drop-in aggregator for debate workflows, parallel RAG pipelines and tool-use ensembles whenever agent trajectories may share upstream documents, tool calls, or message histories. It requires logging document identifiers, citation spans, tool-call signatures or message ancestry; when such logs are unavailable, equal-weight lineage aggregation is the safer fallback. Where all agents already observe the same context, it offers little accuracy gain, and the paper says so.

Q3. Relevance of the theorems to the technique

Theorem 3 is tied explicitly to the method: its characterization "governs Steps 1–2 of Algorithm 1, which remove duplicate trace IDs and group traces by evidence fingerprints" (Sec. 3.2). Sec. 3 closes by stating that the four theorems jointly shape RIDE, and Sec. 3.4 builds the two instantiations on them: controlled oracle diagnostics and non-oracle deployment. This gives the reader the theory-to-method path at the framework level and, for the deduplication and grouping rule, at the step level.

We thank the reviewer again for the careful reading and for comments that have materially improved both the experiments and the presentation. We hope the revision addresses the concerns and would welcome any further suggestions.

Reviewer_i5csOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies a problem in multi-agent LLM systems, where repeated answers based on the same evidence may be treated as independent support. In this scenario, when several agents use the same evidence, voting counts this evidence many times. Then a single agent with new, maybe correct evidence can lose the vote.

Authors introduce term "evidence lineage" and propose a framework RIDE (in two versions: RIDE-CONTROLLED and RIDE-CAL). It groups responses by their evidence sources and avoids giving extra weight to duplicated information.

The paper provides also theoretical results, in the form of theorems and propositions. They cover following questions and topics: on why correlation alone cannot identify reliable consensus, replication invariance, and the distinction between errors caused by missing evidence and errors introduced during evidence handling.

The main contribution of the paper is discussing why it is important to distinguish independent evidence from repeated information when combining answers from multiple agents. Although, the experiments show better calibration and robustness to exact replication, the improvements in accuracy are still less convincing.

Critical strengths
  1. Relevant, practical problem and clear theoretical formulation. The paper investigates valid practical problem: agreement between agents does not always mean independent evidence. The authors show that even calibrated confidence is not enough without provenance information. They also make a useful distinction between failures in acquiring evidence and errors in handling it. I think this is the main strength of the paper, as it helps to better understand the limits of multi-agent consensus.

  2. Useful analysis of the role of evidence. The factorial study measures the effect of answer-bearing evidence and of distractors separately. It is shown that answer-bearing evidence has much bigger influence on accuracy than distractors. This supports the view that access to relevant information is a key factor for reliable multi-agent systems. I consider this point valuable, since many existing approaches concentrate mainly on the way how agents discuss or combine their answers.

  3. Good calibration results, careful experimental design and then evaluation. RIDE-CAL gives clearly better calibration than confidence weighting (NLL drops from 1.646 to 0.650). I also like that the parameters were fixed before the test. Authors do not hide weak points (when accuracy gains are not significant).They also tested the method with incorrect provenance information

Critical weaknesses
  1. Limited non-oracle evaluation. The main non-oracle evaluation covers only a small number of tasks from one dataset and one model family. Although RIDE-CAL improves calibration, the accuracy gain over confidence weighting is small (52.0% vs. 50.5%) and not statistically significant. For me, this is an important limitation. We should expect a broader evaluation to support the general claims of the paper.

  2. Unclear benefit over simpler methods. The simpler hard lineage selector gives results close to RIDE-CAL, both in accuracy and calibration. This raises the question of how much the full RIDE-CAL method actually improves the results. In particular, I miss a comparison with a baseline that uses the same duplicate removal and calibration steps (but without the lineage model). In my opinion, such a comparison would be important to understand whether the additional complexity of RIDE-CAL is justified.

  3. The main motivating problem is not sufficiently tested. The paper starts from a clear and practical problem: many agents repeat the same evidence and outvote one agent with independent information. But in the main non-oracle experiment, every lineage has the same number of responses (two). So the situation from the introduction is not directly tested in this experiment. This is an important gap between the problem described in the paper and the cases tested in the experiment.

Questions for authors
  1. How does RIDE-CAL compare with confidence weighting using the same duplicate removal and calibration steps, but without the lineage-based aggregationl?

  2. What would be the results for lineages with different numbers of responses? For example, five to seven different traces from one lineage against one trace from another lineage.

  3. Would a larger evaluation show a statistically significant accuracy gain over confidence weighting?

2 comments
AuthorsOpenReview ↗

Response to Reviewer i5cs

We sincerely thank the reviewer for the time and care invested in reading our submission and for the constructive comments, which have helped us substantially improve the paper. The revision addresses the three weaknesses directly: the non-oracle evaluation is now multi-dataset and multi-model, the motivating unequal-replication case is tested, and the matched "same duplicate removal and calibration" baseline is included.

W1. Limited non-oracle evaluation

The 200-task single-model confirmation is replaced by a full evaluation over three multi-hop QA datasets (HotpotQA, 2WikiMultihopQA, MuSiQue) and five models (DeepSeek-V3.2, Gemma-3-27B-IT, Llama-3.3-70B-Instruct, Mistral-Large-2407, Qwen3.5-Flash), 250 test questions per dataset-model cell: 3,750 task-model units and 22,500 traces (Sec. 4; Appendix A, Table 5). All selectors and calibration maps are fitted on development data and frozen before test evaluation (Appendix A).

Table 1. Full multi-dataset balanced evaluation (paper Table 8, Appendix D.3)

MethodAccuracy (%)NLLBrierECE
Replica-collapse flat35.250.6070.2090.055
Confidence-weighted instances35.250.6070.2090.055
Hard lineage vote33.01–––
Hard lineage selector35.28–––
Source-free RIDE-CAL (no source IDs)35.28–––
RIDE-CAL35.470.6040.2070.065

The statistical unit is the task-model unit (3,750 in total, 15 dataset–model cells). All lineage-aware rows use the same post-merge lineage definition; flat and confidence-weighted rows aggregate individual traces. NLL, Brier, and ECE use development-fitted maps. The accuracy difference between RIDE-CAL and the matched calibrated instance baselines is +0.22 points, and the hard lineage selector is within 0.19 points; the source-free variant, which receives no source identifiers, matches the selector at 35.28%. The paper therefore states non-inferiority on balanced panels and reports per-cell results for all 15 cells (Table 2 below). The claim defended in the paper is the mechanism under unequal replication (Sec. 5.2).

Table 2. Per-cell coverage across datasets and models (paper Table 4, Sec. 5.4)

ModelHotpotQA flatHotpotQA RIDE2Wiki flat2Wiki RIDEMuSiQue flatMuSiQue RIDE
DeepSeek-V3.239.639.243.642.819.621.6
Gemma-3-27B-IT27.226.822.823.614.415.6
Llama-3.3-70B59.659.652.853.225.225.6
Mistral Large 240742.042.839.638.019.620.4
Qwen3.5-Flash51.250.849.650.022.022.0
Overall35.2535.47

Each cell is 250 test questions. Cell-level differences alternate in sign and lie within sampling noise; the overall rows are the weighted averages. The table is included to show that the evaluation is not concentrated in one dataset or one model, and it supports the paper's statement that balanced-panel accuracy is on par with flat voting.

W2. Benefit over simpler methods

We agree this was the key missing comparison, and it is now included. All methods use the same post-merge lineage definition; flat and confidence-weighted aggregation operate at trace level. Table 1 above reports the comparison: replica-collapse flat 35.25% / NLL 0.607, confidence-weighted instances 35.25% / NLL 0.607, hard lineage selector 35.28%, and RIDE-CAL 35.47% / NLL 0.604.

The value of lineage-level aggregation appears under unequal replication (Sec. 5.2). Table 3 below reports the six-layout confirmation, where the endpoint is whether the method selects the lineage that carries the answer.

Table 3. Response-count imbalance confirmation (paper Table 2, Sec. 5.2)

BatchLayoutRIDE correctInstance correctConfidence correctRIDE wrongYouden's J
A6:2:247.928.139.617.630.3
A2:6:245.130.543.914.330.8
A2:2:652.232.642.415.037.2
B6:1:159.822.444.922.237.6
B1:6:162.929.953.624.838.1
B1:1:660.630.844.922.837.7

Layouts are written as a:b:c = numbers of sampled traces on the lexical, dense, and complementary-PRF retrievers, so 6:1:1 means one source is replicated six times while the others are sampled once. Correct is the share of eligible tasks in which the method selects the unique smallest post-merge lineage that contains the gold answer; wrong is the share in which it selects the unique smallest lineage carrying a wrong answer. Eligible tasks are those where exactly one such lineage exists. Batch A and Batch B are independent confirmation batches with different frozen task pools; their balanced-panel accuracies are 35.1% and 39.6%, and non-inferiority is computed within each batch. Across all six layouts RIDE-CAL selects the correct minority lineage in 45–63% of cases versus 40–54% for confidence-weighted voting and 22–33% for unweighted voting, while accuracy remains within 2 points of the batch baseline.

W3. The motivating case is not tested

This is now the centrepiece of the evaluation. Table 3 above reports six unequal-replication layouts in two independent confirmation batches. The layouts include the case the reviewer names (six traces from one lineage against one from each of the others: 6:1:1, 1:6:1, 1:1:6) as well as milder imbalance (6:2:2, 2:6:2, 2:2:6). In every layout RIDE-CAL recovers the correct minority lineage far more often than instance-level voting, and accuracy stays within the 2-point non-inferiority margin.

Q1. RIDE-CAL vs. confidence weighting with the same deduplication and calibration

The matched baselines in Table 1 are replica-collapse flat voting (duplicate removal followed by instance voting) and confidence-weighted instances, both with development-fitted calibration; each reaches 35.25% accuracy and NLL 0.607. RIDE-CAL reaches 35.47% and NLL 0.604. On the imbalanced panels in Table 3, confidence-weighted voting recovers the correct minority lineage in 40–54% of cases versus 45–63% for RIDE-CAL, while RIDE-CAL's incorrect-minority rate is higher (14.3–24.8%), which is why both endpoints are reported.

Q2. Lineages with different numbers of responses

Answered by the six layouts in Table 3, including 6:1:1. Sec. 5.5 also quantifies how decision mass responds to trace count: the voting-share slope is −0.0007 for RIDE-CAL, 1.000 for instance majority, and 0.919 for confidence-weighted voting, i.e., instance-level methods accumulate weight linearly with replication while RIDE-CAL keeps each lineage's weight invariant.

Q3. Would a larger evaluation show a significant accuracy gain

The larger evaluation supports non-inferiority, and the paper states its claims at that level: the mechanism results concern minority recovery under unequal replication (Table 3), not a balanced-panel accuracy gain (Table 1).

We thank the reviewer again for the careful reading and for suggestions that directly shaped the revision, and we hope the new evaluation addresses the concerns raised.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.