acceptodds
Under review as a conference paper at ICLR 2027

COS-WEB: SUSPICION-DRIVEN MULTI-AGENT REASONING WITH STRUCTURED COMMUNICATION

Abstract

Large language models (LLMs) are increasingly used for complex reasoning, but a single reasoning path can remain vulnerable to missing knowledge and internally consistent errors. LLM-based multi-agent systems (LLM-MAS) address this limitation through information exchange and collaborative decision-making. MultiAgent Debate (MAD) lets agents critique reasoning paths and revise judgments over multiple rounds. However, agreement on an answer does not guarantee that its supporting rationale is reliable: when multiple agents share an unverified assumption, their interaction may further reinforce a false consensus. Meanwhile, existing communication optimization methods focus mainly on interaction structures, messages, or participant selection. They rarely represent unresolved issues explicitly to guide local communication and subsequent re-examination. To address this, we propose CoS-Web (Chain-of-Suspicion with Spider-Web Communication), a multi-agent reasoning framework that connects suspicion-driven interaction with subsequent communication through structured unresolved issues. Chain-of-Suspicion (CoS) is a suspicion-driven interaction protocol that progressively examines agent judgments, their anticipated defenses, and collective reasoning risks to formulate structured unresolved issues. Spider-Web Communication uses these issues to organize local Ring Communication among relevant agents on top of Radial global-state sharing, and re-examines whether the issues remain unresolved after communication. Experiments and analyses across multiple reasoning benchmarks show that CoS-Web improves overall reasoning accuracy while balancing communication costs, and handles cases in which multiple agents share erroneous assumptions more robustly

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models Multi-Agent Systems Multi-Agent Debate Agent Communication Reasoning Verification

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_T9U5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CoS-Web, a multi-agent reasoning framework designed to address false consensus. Its COS component examines agents’ judgments to identify unresolved reasoning issues. These issues are then used by Spider-Web Communication to select relevant agents for local discussion. After each communication round, a judge checks whether the issues have been resolved and decides whether further interaction is needed.

Critical strengths

The paper aims to address false consensus problem of multi-agent system, where multiple agents agree on an answer based on the same unsupported assumption.

The paper proposes COS component to identify unresolved reasoning issues. These issues are then used by Spider-Web Communication to select relevant agents for local discussion.

Critical weaknesses
  1. It is not clear whether D1 and D2 use the same agent and context that generate original answer, or trigger new agent calls. If D1 and D2 use the same agent (as shown in line 210-211: “Direct Suspicion (D1) performs individual scrutiny of other agents’ publicly visible judgments.”), there is no independent source for correcting a shared false assumption. Then criticizing and exchanging the same internal knowledge may not reliably correct the shared errors. If D1 and D2 does not trigger new agent calls, the method seems to encourage reconsideration and selective multi-agent debate.

  2. The function of D2 and how it can evaluate residual risk are unclear. The anticipated defense is not generated by the criticized agent, and D2 does not receive any additional evidence. It is unclear how D2 can improve the reliability of the identified suspicions. The ablation study also does not demonstrate the effectiveness of D2. I suggest adding a D1+D3 variant and comparing it with the full D1+D2+D3 to isolate the contribution of D2.

  3. The experimental evaluation is incomplete. The paper only evaluates Qwen3-14B-AWQ. It is unclear whether the findings generalize to other models or model families. In addition, the paper does not report the number of agents used for the multi-agent baselines. For a fair comparison, these baselines should use the same number of reasoning agents as the proposed method.

Questions for authors
  1. Can you clarify whether D1 and D2 are performed by the same agent and within the same context used to generate the initial answer, or by independently instantiated critic agents? If the same agents are reused, what mechanism enables CoS-Web to correct an erroneous assumption shared by all agents, and how does this differ from existing self-refinement and sparse multi-agent debate methods?

  2. How does D2 improve the reliability of the identified? Can you add a D1+D3 ablation to isolate the contribution of D2?

  3. Do the results generalize beyond Qwen3-14B-AWQ? Also, how many agents are used for each multi-agent baseline, and are they configured with the same number of reasoning agents as CoS-Web?

Reviewer_uwUDOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

CoS-Web is a multi-agent reasoning protocol in which five role-prompted agents, all running the same backbone (Qwen3-14B-AWQ), share compact states through a Judge. They then execute a three-stage Chain-of-Suspicion: each agent critiques the others, each agent anticipates defenses to its own critiques, and the Judge consolidates these into "unresolved issues", each tagged with the agents involved. Agents named in an issue exchange disclosures in a local ring, and the Judge rechecks which issues remain, iterating for up to 3 rounds. On GSM8K, MATH-500, GPQA-D, and StrategyQA, CoS-Web reaches 83.3% macro accuracy, versus 81.3% for the best baseline. It also holds up better when 1-3 agents are seeded with a shared wrong assumption.

Critical strengths
  • The motivation is apt: agreement does not imply correct reasoning, and shared-assumption errors are an under-addressed failure mode of debate.
  • All baselines are reimplemented under a shared backbone with shared initial responses, enabling paired comparisons. Paired bootstrap CIs are reported, and the paper honestly notes when the CIs include zero.
  • A useful set of controls is present: generic-critique, matched-budget routing, and same-prompt agents.
Critical weaknesses

1. The headline gain rests mostly on StrategyQA, where the baselines behave anomalously. On StrategyQA, Majority Vote over the five agents (70.6) and MAD-Full (71.0) score well below single CoT (76.5) and CoT-SC (78.2). A majority vote over five reasoning paths should not lose ~6–8 pp to a single path, and CoT-SC (also 5 votes) does not. The same-prompt variant in Table D.1 still gives MAD-Full 69.8, so the role prompts alone are not the cause. This points to a systematic problem in the multi-agent pipeline on this dataset. Candidates include the five-field JSON state, the strict "FinalAnswer: Yes/No" parser, or the role-conditioned initial responses that all multi-agent methods share. CoS-Web then "recovers". Since StrategyQA alone drives most of the 2.0 pp macro gain and the whole "largest gains where examining assumptions matters" narrative, the StrategyQA result cannot be read as evidence for the method.

2. The compute comparison is incomplete, and the token accounting looks implausible. CoS-Web uses about 13.3k tokens per question, versus about 7k for CoT-SC and 10.3k for Free-MAD-N, and it makes far more LLM calls per round. Yet no compute-matched single-agent baseline is reported, such as CoT-SC with 10-20 samples, or self-consistency plus an LLM judge at equal tokens. The reported costs also look low for Qwen3 with thinking mode enabled and a 16k output limit: 1.6k tokens per GPQA-D question for CoT and 1.1k per initial agent response on GSM8K. Please clarify whether thinking tokens are counted. If they are not, all efficiency statements, including Fig. F.5 and the "balancing communication costs" claim in the abstract, are invalid.

Questions for authors
  1. Why does the five-agent majority vote fall below single CoT on StrategyQA? What are the parse-failure rates per method there?
  2. Are Qwen3 thinking tokens included in all Tokens/Q numbers?
  3. How were the 490 StrategyQA training items selected?
  4. In the "no Final Judge" ablation (Table C1), how is the final answer produced?
Reviewer_bPz7OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CoS-Web is a structured multi-agent discussion framework aimed at reducing shared reasoning errors and false consensus. Agents identify unsupported claims, consider hypothetical defenses, and aggregate remaining concerns into explicit unresolved issues. These issues guide local communication among selected agents, after which a Judge rechecks their status and decides whether discussion should continue. The paper evaluates the framework on four reasoning benchmarks, primarily using Qwen3-14B-AWQ, with an additional Llama-3.1-8B-Instruct comparison.

Critical strengths

The paper has a reasonable motivation: agreement among agents does not establish that the underlying reasoning is reliable. Explicitly recording unresolved issues and using them to organize subsequent discussion gives a concrete procedure for investigating this problem. I found this aspect interesting and useful. My main reservation concerns whether the empirical evaluation establishes the significance and effectiveness of the proposed solution as underlying model capability improves.

Critical weaknesses
  1. The continued importance of the targeted problem is not sufficiently established across model capability levels. The evaluation centers on Qwen3-14B-AWQ and Llama-3.1-8B-Instruct. These experiments do not establish whether shared false consensus remains frequent and consequential with substantially stronger reasoning models, or whether much of the motivating difficulty reflects the limitations of the tested backbones. This directly affects my assessment of the importance of the problem being solved. I am not assuming that stronger models eliminate shared errors; I would like evidence clarifying their prevalence and impact as capability improves.

  2. The evidence for a capability-robust advantage of the protocol is limited. Appendix I.1 is a useful cross-backbone check and shows gains over MAD-Full on GPQA-Diamond and StrategyQA using Llama-3.1-8B-Instruct. However, it does not repeat the closest-baseline comparisons, such as Free-MAD-N, on that backbone, and two medium-scale model settings do not characterize how the benefit changes with capability. Earlier work, including Wang et al. (ACL 2024), Rethinking the Bounds of LLM Reasoning: Are Multi-Agent Discussions the Key?, highlights the importance of strong single-agent prompting and backbone choice. Although CoT and self-consistency baselines are included in the main experiments, it remains unclear whether the proposed discussion procedure retains a meaningful advantage with substantially stronger models and strong prompting under comparable inference budgets.

These empirical questions motivate my current weak-rejection recommendation. The paper is generally clear, but Figure 1 would benefit from layout refinement: small annotation text and crowded labels make the overview less readable. This is a specific presentation improvement rather than my main reason for rejection.

Questions for authors
  1. Does shared false consensus remain a substantial failure mode with stronger reasoning backbones? Please assess how often it occurs and how consequential it is as model capability increases. This evidence would affect my judgment of the problem's significance, beyond showing transfer to another model family.

  2. Does CoS-Web retain its advantage with stronger backbones against the closest competing discussion protocols and well-prompted single-agent or self-consistency baselines under comparable inference budgets? The existing second-backbone comparison against MAD-Full is useful, but does not resolve this question.

  3. If the intended contribution primarily concerns resource-constrained, medium-scale models, can you substantiate the importance and cost-benefit trade-off in that setting, including comparison with allocating the budget to a stronger single model? Evidence supporting either continued relevance at stronger capability levels or a compelling resource-constrained use case could change my assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.