- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the risks posed by false access labels in LLM multi‑agent aggregator scenarios. An access label indicates whether a sub‑agent member has obtained the information required to answer a question, rather than labelling whether an answer is correct. The authors design controlled experiments where the full response texts of sub‑agent members are kept fixed, while only the access labels are swapped. The experiments are conducted with five aggregator models across multiple‑choice benchmarks (MMLU‑Pro, SuperGPQA) and multi‑hop document QA datasets (2WikiMultiHopQA, MuSiQue).
Critical strengths
It features a rich set of task types, covering both multiple‑choice question (MCQ) and multi‑hop document question‑answering paradigms.
It explicitly points out that matching a member’s answer does not equate to answering correctly. Particularly in document‑based tasks, members without access to supporting evidence may still arrive at the correct answer by chance. The metric design is well‑aligned with task‑specific characteristics.
Critical weaknesses
The paper repeatedly acknowledges that the underlying mechanism remains unknown and only presents observational phenomena. It cannot disentangle whether aggregators treat labels as simple heuristic shortcuts, struggle to reconcile labels with textual evidence, or suffer interference from negative labels. Diagnostic experiments targeting these root causes are absent.
In multiple‑choice questions, the evidence‑holding member is the only model that can legitimately select the correct option. By contrast, in document‑based tasks, members without supporting evidence can still guess the correct answer. As a result, the definition of harm caused by false labels is inconsistent across the two task categories, yet the paper fails to develop a unified metric for quantifying such harm.
The main experiments employ long‑text access labels with a fixed format. In real‑world systems, meta‑labels can take diverse forms such as compact JSON tags, embedded metadata, and abbreviated markers. Only limited small‑scale ablation tests on label rephrasing are conducted, leading to limited test coverage.
Questions for authors
In these experiments, member models have no visibility into the access labels. In real‑world multi‑agent systems, if members themselves can also observe access labels, their outputs may change. Would the overall system risk induced by false access labels then increase or decrease?
When the aggregator receives outputs from a large number of members simultaneously (e.g., 8‑10 member models), will the misleading effect of false access labels be amplified or diluted? The paper only tests settings with 3‑5 members. Can these observed patterns be generalized to such scenarios?
3 comments
Dear Reviewer uS3n,
Thank you for your review. We would appreciate hearing whether our response addresses your concerns, and are happy to clarify anything further.
Best regards,
The Authors
Dear Reviewer uS3n,
As the discussion closes today, we would like to summarise how the revision answers your five points:
- W1, root cause. The diagnostic you asked for supports your third candidate. Most of the loss comes from the false "did NOT" on the member that had the answer: 15.6–30.0 points for all five aggregators.
- W3, label format. A compact JSON tag misleads as well: matching rises by 20.4–60.4 points for all five.
- Q2, 8 to 10 members. With ten members the loss is 43.6 and 50.0 points for the two aggregators tested, against 42.4 and 52.0 with four.
- W2 and Q1. The matching rate is the measure common to all tasks, and members that see the labels are now named as a limitation.
Three of the new experiments in the revision come from your review, and we are grateful for them. If these answers resolve your concerns, we would appreciate it if you would update your rating.
Best regards,
The Authors
Response
Thank you for the careful review and for the specific experiments you suggested. We ran three of them. Line numbers below refer to the revised PDF.
W1. Which root cause? We ran the diagnostic. Most of the loss comes from one negative label, the false "did NOT" on the member that had the answer: 15.6–30.0 points for all five aggregators. The false "DID" label alone costs 3.2–14.0 points. This points to your third candidate, interference from negative labels, with a smaller pull from the positive label. The evidence is behavioural; an internal mechanism is beyond it. The table is in result 3 of our general comment; in the paper, see Sec. 3.3 (lines 424–431), Fig. 7, and App. D.2.
W2. A unified metric. Matching is the common measure. It asks the same question on every task: does the aggregator give the falsely labelled member's answer? Swapped labels raise it on all five datasets. Accuracy differs by task for the reason you give, since a matched answer is always wrong on multiple choice and can be right on documents; we therefore report it per task. See Sec. 2 (lines 180–182).
W3. Label format. We tested a compact tag,
{"has_correct_option": true}. It raises matching by 20.4–60.4 points over no label for all five aggregators. See Sec. 3.1 (lines 328–330) and App. B.6.Q1. Members who see the labels. This setting is outside our design: members never see the labels, so their responses stay fixed and any change comes from the label alone. We name it as a limitation and an open question. See the Limitations (lines 532–533).
Q2. Eight to ten members. We ran ten. Loss (percentage points, MMLU-Pro STEM):
The effect is neither amplified nor diluted. The six added responses are further answers from the three non-holder models. See Sec. 3.1 (lines 327–328) and App. B.9.
Closing. Thank you again for the suggestions, which made the paper more complete. If our answers address your concerns, we kindly ask you to reconsider your score. We are glad to discuss any remaining point.