- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper argues that multi-agent LLM systems overcount agreement when several reports share upstream evidence. It formalizes this with evidence lineages, proves that correlation and calibrated reliabilities cannot identify the right aggregate without dependence information, and characterizes aggregators that ignore added replicas. The proposed method, RIDE-CAL, removes duplicate traces, groups the rest by evidence fingerprint, and combines lineages with a calibrated reliability model. On 200 HotpotQA tasks with three retrievers, it reaches 52.0% accuracy against 50.5% for confidence-weighted voting, lowers answer NLL from 1.646 to 0.650, and never changes its output under 3,600 exact-replication attacks.
Critical strengths
-
The paper studies a practical. Votes that share upstream evidence are counted as independent, so a minority with new evidence loses (Appendix D.3).
-
The evaluation protocol is disciplined. Parameters are fixed on development data before the confirmation run. The paper reports paired tests and states that its accuracy gain is not significant.
-
The provenance-corruption study shows that false merges hurt accuracy while false splits do not, which is a practical rule for building lineage labels.
Critical weaknesses
1. My main concern is that the 0% replication result holds by construction and misses the motivating case.
A replica is a copy made "without a new model call" (line 208), so Theorem 3 does not cover the motivating example of three agents that read the same context (line 42). Algorithm 2 also deletes records with a repeated trace identifier (line 816). If the injected copies keep their identifiers, any method with this step scores 0%, yet the paper does not report the audit for its own exact-replica collapse baseline. The relevant test uses near-replicas, such as new samples from the same context or copies with new identifiers.
2. The non-oracle study has no replication imbalance to correct.
Each retriever contributes exactly two traces (line 323), and exact-replica collapse matches instance majority on every metric (Table 5). Against confidence-weighted voting, RIDE-CAL gains 1.5 points (95% CI −0.5 to 4.0), and without document provenance it falls to 48.5% (line 430), below instance majority.
3. The calibration gain appears where all sources agree, so lineage does not explain it.
It comes from the 85 tasks without source conflict (NLL 2.991 to 0.637, line 400), while on the 115 conflict tasks RIDE-CAL is no better than confidence weighting (0.659 vs. 0.651). RIDE-CAL fits two Platt maps on development data (lines 785–786), and the paper describes no such calibration for confidence weighting, so that baseline needs the same calibration.
4. The controlled studies rely on an oracle.
Adding the paragraph that the design defines as the only answer-bearing evidence raises accuracy by 34 to 43 points (Table 1), which is expected. The handling contrasts compare voting with a rule that routes to the source known to hold the gold support (line 260), and without that oracle, one shared lineage against one independent lineage is a tie.
5. The theory is mostly known, and the paper compares with no dependence-aware method.
Theorem 1 is the correlated Condorcet result (Ladha, 1992), Theorem 2 is standard double counting, Theorem 3 follows from the definition, and Theorem 4 specializes Ao et al. (2026) by the paper's own account (line 242). No experiment estimates the acquisition and handling terms. The paper cites source-copying detection (Dong et al., 2009) and CAGE-CAL (Huang et al., 2026) but uses neither as a baseline, and it omits the surprisingly popular algorithm (Prelec et al., 2017), which targets the same failure of a correct minority.
6. The evidence is narrow for claims about multi-agent reasoning.
The non-oracle results come from one 200-task HotpotQA set with retrieval ensembles of a single, unnamed Qwen model, and agent communication such as debate appears only in the oracle studies. Lineages are formed by hashing the rendered context (line 817), not by the answer-relevant evidence of Definition 1.
Questions for authors
-
Do the injected copies keep the trace identifier that Algorithm 2 removes, and what do the exact-replica collapse baseline and RIDE-CAL show when the added trace is a new sample from the same context?
-
Can the authors run a non-oracle experiment with unequal replication, for example six traces from one retriever and one from each of the others?
-
What NLL do instance majority and confidence weighting reach after the same development-fitted Platt map?
-
What accuracy does RIDE-CAL reach in the controlled setting without the designated-source rule?
-
How does RIDE-CAL compare with source-copying detection (Dong et al., 2009) or the surprisingly popular algorithm (Prelec et al., 2017)?
6 comments
W3. Calibration by conflict status and matched calibration
The comparison has been rebuilt. Calibration is stratified by whether post-merge lineage representatives disagree, and every method's probabilities use development-fitted Platt maps.
Table 4. Calibration by post-merge conflict status (paper Table 3, Sec. 5.3)
| Stratum | Method | n | Accuracy (%) | NLL | Brier | ECE-10 |
|---|---|---|---|---|---|---|
| Conflict | Calibrated flat | 2,223 | 23.6 | 0.551 | 0.183 | 0.091 |
| Conflict | RIDE-CAL | 2,223 | 24.1 | 0.548 | 0.181 | 0.086 |
| Non-conflict | Calibrated flat | 1,527 | 52.2 | 0.689 | 0.247 | 0.030 |
| Non-conflict | RIDE-CAL | 1,527 | 52.2 | 0.684 | 0.245 | 0.039 |
Conflict panels are those where post-merge lineage representatives disagree; non-conflict panels are those where they agree. All maps are fitted on development data only. The differences between the two methods are small in both strata, and the paper presents calibration as a supporting stability result. The baselines in Table 6 also use development-fitted maps, as its caption states.
W4. The controlled studies and the oracle setting
The controlled setting is described as "an oracle setting where passage roles are known to the experimenter" (Sec. 3.4), and Sec. 5.6 compares rules on "identical controlled trace multisets" to isolate handling. The controlled study serves as a mechanism diagnostic, while the deployable evidence comes from the non-oracle full evaluation (Sec. 5.4) and the imbalance confirmations (Sec. 5.2). As a source-free check, removing source-identity features from the selector and refitting on development leaves accuracy essentially unchanged (35.28% versus 35.47%; Appendix D.3, Table 8).
W5. Theoretical positioning and dependence-aware baselines
We agree with the characterization: Theorem 1 restates the correlated Condorcet result (Ladha, 1992); Theorem 2 is the double-counting argument for shared evidence; Theorem 3 is a characterization that follows from the definition of replica equivalence; Theorem 4 specializes the decomposition of Ao et al. (2026). The paper presents these results as the framework that fixes the object of study and yields testable predictions, and its claims rest on the empirical mechanism study.
Two additions respond to this weakness. First, Sec. 5.5 reports empirical decomposition under 0/1 task loss relative to a three-source reference: acquisition gaps of 0.042 (one source) and 0.010 (two sources), and a handling gap of −0.003; the paper labels this as a proxy estimate. Second, Appendix D.4 adds dependence-aware baselines.
Table 5. Acquisition and handling gaps (paper Sec. 5.5)
| Quantity | One source | Two sources | Three sources |
|---|---|---|---|
| Acquisition gap (0/1 task loss) | 0.042 | 0.010 | 0 |
| Handling gap (0/1 task loss) | – | – | −0.003 |
The acquisition gap measures the loss attributable to evidence that was not acquired, holding the aggregation rule fixed: it shrinks as more sources are observed and reaches zero with all three. The handling gap is the residual loss after all acquired traces are in the transcript and is close to zero, which is the empirical analogue of the acquisition–handling decomposition in Theorem 4. This is a proxy estimate under 0/1 task loss, as Sec. 5.5 states.
Table 6. Dependence-aware baselines (paper Table 10, Appendix D.4)
| Method | Accuracy (%) | NLL | Brier | ECE-10 | Additional calls |
|---|---|---|---|---|---|
| Source-copy | 33.09 | 0.578 | 0.196 | 0.021 | 0 |
| CAGE-CAL | 33.81 | 0.593 | 0.203 | 0.023 | 0 |
| Surprisingly-popular (Prelec) | 26.13 | 0.497 | 0.166 | 0.028 | 0 |
| RIDE-CAL | 35.47 | 0.604 | 0.207 | 0.065 | 0 |
All rows use the same 22,500-trace pool, so no additional calls are needed. NLL and Brier use each method's development-fitted maps. RIDE-CAL optimizes for replication invariance and minority-evidence recovery, not for NLL minimization on balanced panels. The three dependence-aware baselines, after their own calibration, achieve lower NLL but substantially lower accuracy (−1.7 to −9.3 points). This reflects a design choice: RIDE-CAL trades calibration tightness on consensus tasks for correctness on minority-evidence tasks (Sec. 5.2). On the conflict stratum (Table 3), where lineage matters most, RIDE-CAL attains NLL 0.548 versus 0.551 for calibrated flat.
W6. Scope of the evidence, lineage resolution, and debate
The evaluation covers three datasets and five models (3,750 test units, 22,500 traces), with per-cell results in Table 4 and the model roster in Appendix A. Lineage resolution is tested directly (Sec. 5.4): fingerprint, document and sentence-level lineages give 35.5% and 35.7% accuracy, so the qualitative result does not hinge on one granularity. Lineage is operationalized by hashing the rendered context, which approximates the answer-relevant evidence state of Definition 1; Sec. 5.4 and Appendix D.6 report how this proxy behaves under false merges, false splits and overlap grouping. Debate appears in the controlled study (Appendix E), and Sec. 6 states the scope of that evidence.
Questions
- Identifiers and near replicas. Exact replicas keep their identifiers and are removed by Step 1; the paper reports this as a construction check. For new samples from the same context, Table 1 reports a 26–29% modal-answer change while Table 2 shows that RIDE-CAL's voting share stays count-invariant (slope −0.0007). Appendix E reports the controlled replication attacks.
- Non-oracle unequal replication. Six layouts in two batches (Table 3).
- NLL after the same development-fitted Platt map. Instance majority and confidence weighting both reach 0.607; RIDE-CAL reaches 0.604 (Appendix D.3, Table 8).
- Controlled setting without the designated-source rule. Without privileged roles the controlled rule reduces to no-lineage voting; Appendix E reports that condition and notes that its contrast with independent-only is a handling diagnostic. In the non-oracle setting, removing source-identity features gives 35.28% versus 35.47%.
- Dependence-aware comparison. Source-copy and surprisingly-popular baselines are reported in Table 6 with the caveats above.
Summary
The paper's contributions are the measurement of source-level versus lineage-level replication (48.0% of panels merge at least two nominal sources), the near-replica and count-invariance analysis (Tables 1–2), and the minority-recovery mechanism under unequal replication (Table 3), with the merged-lineage boundary discussed in Sec. 7.
We thank the reviewer again for the careful and technically specific reading; the points raised have directly shaped the revision, and we would be glad to address any remaining concerns in the discussion.
Response to Reviewer 8E9b
We sincerely thank the reviewer for the time and care invested in this detailed, technically specific review, and for the constructive comments that have helped us improve both the experiments and the presentation. We address each weakness and question below, with the corresponding tables.
W1. The 0% replication result and the motivating case
The paper reports the exact-replica result as a construction check: Step 1 of Algorithm 1 removes repeated trace identifiers, so an exact replica cannot change the output (Sec. 5.5). The case that matters — near replicas with fresh trace identifiers — is measured directly in Sec. 5.5.
Table 1. Same-evidence resampling on 200 tasks (paper Sec. 5.5)
Two independent trace subsets are drawn from the same per-source contexts, and the modal answers of the two subsets are compared. Agreement of 0.735 means that on 26.5% of panels a fresh sample from the same evidence changes the modal answer. The table therefore quantifies the stochastic drift that exact-replica invariance does not cover, and it is the empirical counterpart of the reviewer's near-replica case.
Table 2. Count sensitivity of decision mass (paper Sec. 5.5)
The slope measures how much extra decision weight a method gives a lineage as its trace count grows; zero means the method is count-invariant. Instance-level methods accumulate weight linearly with replication, whereas RIDE-CAL's lineage weight does not grow with the number of resamplings. Table 8 in the appendix also reports the replica-collapse baseline (35.25% accuracy, NLL 0.607); on the frozen bank it coincides with flat aggregation. Appendix E reports the controlled replication attacks: instance voting changes in 58.5% of 1,600 attacks, RIDE-CONTROLLED in 0%.
W2. The non-oracle study and replication imbalance
This is now the main non-oracle experiment. Sec. 5.2 reports six unequal-replication layouts in two independent confirmation batches.
Table 3. Response-count imbalance confirmation (paper Table 2, Sec. 5.2)
Layouts are
a:b:c= numbers of sampled traces on the lexical, dense, and complementary-PRF retrievers. Correct is the share of eligible tasks where the method selects the unique smallest post-merge lineage containing the gold answer; wrong is the share where it selects the unique smallest lineage carrying a wrong answer. Batch A and Batch B are independent confirmation batches with different frozen task pools and balanced accuracies of 35.1% and 39.6%; non-inferiority is computed within each batch. Across all six layouts RIDE-CAL selects the correct minority lineage in 45–63% of cases versus 40–54% (confidence) and 22–33% (unweighted), with accuracy within 2 points of the batch baseline.