- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposes Conditional Progressive Pruning (CPP), a learned controller for heterogeneous multi-agent debate. CPP sequentially decides which earlier responses each agent should receive, conditioning later pruning choices on selected responses and prior debate rounds. A FuseGLU architecture combines the current query, candidate responses, and Agent Behavior Embeddings extracted from responses. The controller is trained with policy gradients using correctness-transition rewards and discounted returns over related actions. On sampled MATH-Hard, GPQA-Diamond, and MMLU-Pro tasks, experiments compare CPP with debate, self-improvement, and scaled consistency baselines using model-dependent inference costs. Additional experiments examine reward design, behavior embeddings, adversarial conditions, output truncation, and alternative model configurations. Reported results show modest accuracy improvements with adjustable accuracy–cost trade-offs.
Critical strengths
-
Conditioning pruning choices on actual intermediate responses and previously selected messages is a reasonable way to exploit information arising during debate. The approach goes beyond static topology built solely from model characteristics.
-
Empirical comparisons include closely related learned methods such as ARG-Designer and AnyMAC, as well as scaled consistency baselines. Accounting for heterogeneous model costs is more informative than comparing raw token counts alone.
Critical weaknesses
-
The paper is difficult to read. The layout is overly dense, with long blocks of text, insufficient structure, and a lack of clear logical progression.
-
The claim of outperforming consistency methods comprehensively “at strictly identical cost” exceeds the evidence. On MMLU-Pro, CPP achieves 66.33% at 99.43% normalized cost, while Consistency-Fuse achieves 66.17% at 78.94% cost. Budget–performance curves and paired significance analysis would better show whether such a small accuracy gain warrants the extra computation.
-
Each main test set contains only 100 questions, and the principal configuration uses three fixed models and three debate rounds. Scaling experiments rearrange these models rather than testing substantially larger teams. Although variability is reported, several gains are small relative to the stated standard deviations, limiting claims of broad superiority and scalability.
-
Important training and cost-accounting details remain insufficiently specified. For correctness-transition rewards, the paper should clarify how intermediate answers are generated and rewards assigned after each edge decision, and report rollout costs, model pricing, encoder configuration, and final hyperparameters. The dependency table in Appendix A does not fully describe the training procedure.
-
The key autoregressive conditioning mechanism is not cleanly isolated. Table 6 studies embeddings, fusion structure, and rewards, but neither removes dependence on previously selected messages under fixed architecture and training budgets nor distinguishes within-round from cross-round conditioning.
Questions for authors
See Weaknesses.
First, we really appreciate your valuable comments, and we apologize for the poor quality of the writing in current version. We consider all our experimental results very solid, but the writing does have problems. The responses below are meant for productive academic discussion rather than as a rebuttal.
Response 1:
Response 2: